[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:11:40 dist-test-slave-x62p krb5kdc[9694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:11:40 dist-test-slave-x62p krb5kdc[9694](info): set up 2 sockets
Oct 23 20:11:40 dist-test-slave-x62p krb5kdc[9694](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251023 20:11:42.193310 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20251023 20:11:42.492499 9687 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.299s user 0.001s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20251023 20:11:42.814668 9687 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.322s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20251023 20:11:43.064129 9687 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.249s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/test-user.keytab.
W20251023 20:11:43.276363 9687 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.212s user 0.000s sys 0.002s
Oct 23 20:11:43 dist-test-slave-x62p krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:11:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:43.346040 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:46293
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:46293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:43.438418 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:43.438652 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:43.438691 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:43.440214 9710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:11:43.440267 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:43.440292 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:43.440312 9710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:11:43.440333 9710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:11:43.442065 9710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:46293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:46293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:43.442345 9710 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:43.442561 9710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:43.445365 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:43.445371 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:43.445510 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:43.445595 9710 server_base.cc:1047] running on GCE node
I20251023 20:11:43.445890 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:43.446146 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:43.447319 9710 hybrid_clock.cc:648] HybridClock initialized: now 1761250303447297 us; error 30 us; skew 500 ppm
Oct 23 20:11:43 dist-test-slave-x62p krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250303, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:43.450459 9710 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:11:43.451128 9710 webserver.cc:492] Webserver started at http://127.9.117.254:40461/ using document root <none> and password file <none>
I20251023 20:11:43.451333 9710 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:43.451372 9710 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:43.451472 9710 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:43.452517 9710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "f89b11d22155449f87ecdb1943a2a4ad"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "ceba704e6ed2db79f0d627965379aa89"
server_key_iv: "44b34d3c7da2689f7b098e9e8db96f88"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.452903 9710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "f89b11d22155449f87ecdb1943a2a4ad"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "ceba704e6ed2db79f0d627965379aa89"
server_key_iv: "44b34d3c7da2689f7b098e9e8db96f88"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.454192 9710 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:11:43.454959 9725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:43.455165 9710 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:11:43.455237 9710 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "f89b11d22155449f87ecdb1943a2a4ad"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "ceba704e6ed2db79f0d627965379aa89"
server_key_iv: "44b34d3c7da2689f7b098e9e8db96f88"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.455308 9710 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:43.474994 9710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:43.477451 9710 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:43.477574 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:43.482093 9710 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:46293
I20251023 20:11:43.482128 9777 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:46293 every 8 connection(s)
I20251023 20:11:43.482484 9710 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:11:43.483193 9778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:43.487915 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad: Bootstrap starting.
I20251023 20:11:43.489216 9778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:43.489756 9778 log.cc:826] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:43.491299 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad: No bootstrap required, opened a new log
I20251023 20:11:43.491686 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 9710
I20251023 20:11:43.491794 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:11:43.492374 9687 external_mini_cluster.cc:1468] Setting key e4905a6444f8f153dafc0dbc795380a3
I20251023 20:11:43.492947 9778 raft_consensus.cc:359] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89b11d22155449f87ecdb1943a2a4ad" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46293 } }
I20251023 20:11:43.493129 9778 raft_consensus.cc:385] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:43.493167 9778 raft_consensus.cc:740] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89b11d22155449f87ecdb1943a2a4ad, State: Initialized, Role: FOLLOWER
I20251023 20:11:43.493295 9778 consensus_queue.cc:260] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89b11d22155449f87ecdb1943a2a4ad" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46293 } }
I20251023 20:11:43.493383 9778 raft_consensus.cc:399] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:11:43.493417 9778 raft_consensus.cc:493] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:11:43.493474 9778 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:43.494076 9778 raft_consensus.cc:515] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89b11d22155449f87ecdb1943a2a4ad" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46293 } }
I20251023 20:11:43.494189 9778 leader_election.cc:304] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f89b11d22155449f87ecdb1943a2a4ad; no voters:
I20251023 20:11:43.494365 9778 leader_election.cc:290] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:11:43.494421 9783 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:43.494580 9783 raft_consensus.cc:697] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [term 1 LEADER]: Becoming Leader. State: Replica: f89b11d22155449f87ecdb1943a2a4ad, State: Running, Role: LEADER
I20251023 20:11:43.494642 9778 sys_catalog.cc:565] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:11:43.494760 9783 consensus_queue.cc:237] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89b11d22155449f87ecdb1943a2a4ad" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46293 } }
I20251023 20:11:43.495209 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f89b11d22155449f87ecdb1943a2a4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89b11d22155449f87ecdb1943a2a4ad" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46293 } } }
I20251023 20:11:43.495311 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [sys.catalog]: This master's current role is: LEADER
I20251023 20:11:43.495554 9785 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f89b11d22155449f87ecdb1943a2a4ad. Latest consensus state: current_term: 1 leader_uuid: "f89b11d22155449f87ecdb1943a2a4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89b11d22155449f87ecdb1943a2a4ad" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46293 } } }
I20251023 20:11:43.495858 9785 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad [sys.catalog]: This master's current role is: LEADER
I20251023 20:11:43.496002 9790 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:11:43.497004 9790 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:11:43.499061 9790 catalog_manager.cc:1357] Generated new cluster ID: 27009bd27a50425583265ebf038f58fd
I20251023 20:11:43.499141 9790 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:11:43 dist-test-slave-x62p krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:43.506894 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:43.493192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.9.117.254:46293)
1023 20:11:43.493388 (+ 196us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:43.493400 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:43.493567 (+ 167us) server_negotiation.cc:408] Connection header received
1023 20:11:43.495028 (+ 1461us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:43.495034 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:43.495076 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:43.495179 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:43.497372 (+ 2193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.498342 (+ 970us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:43.499901 (+ 1559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.500182 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:43.503004 (+ 2822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:43.503012 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:43.503012 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:43.503037 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:43.505052 (+ 2015us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:43.505507 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:43.505513 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:43.505513 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:43.505586 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:43.505807 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:43.505808 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:43.505808 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:43.506041 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:43.506180 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:43.506427 (+ 247us) server_negotiation.cc:300] Negotiation successful
1023 20:11:43.506566 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20251023 20:11:43.512105 9790 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:11:43.513067 9790 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:11:43.529093 9790 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f89b11d22155449f87ecdb1943a2a4ad: Generated new TSK 0
I20251023 20:11:43.529347 9790 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:43.593811 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46293
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:43.687709 9806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:43.687969 9806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:43.687999 9806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:43.690122 9806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:43.690203 9806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:43.690239 9806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:11:43.692236 9806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:43.692507 9806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:43.692898 9806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:43.696333 9814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:43.696383 9812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:43.696550 9806 server_base.cc:1047] running on GCE node
W20251023 20:11:43.696592 9811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:43.696825 9806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:43.697063 9806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:43.698211 9806 hybrid_clock.cc:648] HybridClock initialized: now 1761250303698197 us; error 30 us; skew 500 ppm
Oct 23 20:11:43 dist-test-slave-x62p krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250303, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:43.700942 9806 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:11:43.701577 9806 webserver.cc:492] Webserver started at http://127.9.117.193:45343/ using document root <none> and password file <none>
I20251023 20:11:43.701779 9806 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:43.701821 9806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:43.701931 9806 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:43.702937 9806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "ac106e985966448e92335655b2191878"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "ab0d267ab2e017126cf6c2c084e4bd71"
server_key_iv: "070b8b74d0ba894e94a6a9f6f6fc9471"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.703238 9806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "ac106e985966448e92335655b2191878"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "ab0d267ab2e017126cf6c2c084e4bd71"
server_key_iv: "070b8b74d0ba894e94a6a9f6f6fc9471"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.704540 9806 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20251023 20:11:43.705457 9821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:43.705687 9806 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:43.705772 9806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "ac106e985966448e92335655b2191878"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "ab0d267ab2e017126cf6c2c084e4bd71"
server_key_iv: "070b8b74d0ba894e94a6a9f6f6fc9471"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.705850 9806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:43.717478 9806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:43.719892 9806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:43.720026 9806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:43.720281 9806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:11:43.720610 9806 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:43.720647 9806 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:43.720674 9806 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:43.720685 9806 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:43.726892 9806 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:36823
I20251023 20:11:43.726958 9934 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:36823 every 8 connection(s)
I20251023 20:11:43.727272 9806 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:11:43.729436 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 9806
I20251023 20:11:43.729530 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:11:43.729718 9687 external_mini_cluster.cc:1468] Setting key 81270c5098ca3d3846dce8eaaece975b
Oct 23 20:11:43 dist-test-slave-x62p krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250303, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:43.737813 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:43.728023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46293 (local address 127.9.117.193:39277)
1023 20:11:43.728211 (+ 188us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:43.728244 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:43.728817 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:43.729062 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:43.729066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:43.729517 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:43.729977 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:43.729981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.730942 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:43.730944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:43.731581 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:43.731584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.731740 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:43.732900 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:43.732919 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:43.734306 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:43.736469 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:43.736477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:43.736480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:43.736716 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:43.736938 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:43.736938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:43.736938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:43.736999 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:43.737278 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:43.737281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:43.737418 (+ 137us) client_negotiation.cc:769] Sending connection context
1023 20:11:43.737498 (+ 80us) client_negotiation.cc:240] Negotiation successful
1023 20:11:43.737634 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20251023 20:11:43.737826 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:43.728056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:39277 (local address 127.9.117.254:46293)
1023 20:11:43.728120 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:43.728122 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:43.728330 (+ 208us) server_negotiation.cc:408] Connection header received
1023 20:11:43.728915 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:43.728916 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:43.728936 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:43.729011 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:43.730143 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.730841 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:43.731686 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.731846 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:43.734405 (+ 2559us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:43.734411 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:43.734412 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:43.734421 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:43.736368 (+ 1947us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:43.736838 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:43.736839 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:43.736840 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:43.736872 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:43.737066 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:43.737067 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:43.737067 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:43.737218 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:43.737295 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:43.737554 (+ 259us) server_negotiation.cc:300] Negotiation successful
1023 20:11:43.737685 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251023 20:11:43.738348 9935 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46293
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:11:43.738485 9935 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:43.738762 9935 heartbeater.cc:507] Master 127.9.117.254:46293 requested a full tablet report, sending...
I20251023 20:11:43.739259 9742 ts_manager.cc:194] Registered new tserver with Master: ac106e985966448e92335655b2191878 (127.9.117.193:36823)
I20251023 20:11:43.740005 9742 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:39277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:43.781045 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46293
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:43.866966 9942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:43.867187 9942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:43.867215 9942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:43.868693 9942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:43.868793 9942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:43.868824 9942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:11:43.870524 9942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:43.870769 9942 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:43.870936 9942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:11:43.874116 9942 server_base.cc:1047] running on GCE node
W20251023 20:11:43.874059 9947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:43.874073 9948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:43.874053 9950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:43.874447 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:43.874675 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:43.875839 9942 hybrid_clock.cc:648] HybridClock initialized: now 1761250303875814 us; error 45 us; skew 500 ppm
Oct 23 20:11:43 dist-test-slave-x62p krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250303, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:43.878810 9942 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:11:43.879495 9942 webserver.cc:492] Webserver started at http://127.9.117.194:45081/ using document root <none> and password file <none>
I20251023 20:11:43.879714 9942 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:43.879757 9942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:43.879871 9942 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:43.880964 9942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "60e6003f1ea74877a278c62b256e4dec"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "20578e0b2d5c61e34640132d59c31564"
server_key_iv: "9f0d343bb84965a869b7c9f3c3fa398c"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.881311 9942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "60e6003f1ea74877a278c62b256e4dec"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "20578e0b2d5c61e34640132d59c31564"
server_key_iv: "9f0d343bb84965a869b7c9f3c3fa398c"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.882720 9942 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:11:43.883648 9957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:43.883908 9942 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:43.883994 9942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "60e6003f1ea74877a278c62b256e4dec"
format_stamp: "Formatted at 2025-10-23 20:11:43 on dist-test-slave-x62p"
server_key: "20578e0b2d5c61e34640132d59c31564"
server_key_iv: "9f0d343bb84965a869b7c9f3c3fa398c"
server_key_version: "encryptionkey@0"
I20251023 20:11:43.884079 9942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:43.893535 9942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:43.895924 9942 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:43.896052 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:43.896281 9942 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:11:43.896611 9942 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:43.896649 9942 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:43.896684 9942 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:43.896715 9942 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:43.904676 9942 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:33233
I20251023 20:11:43.904887 10070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:33233 every 8 connection(s)
I20251023 20:11:43.905189 9942 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:11:43.905943 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 9942
I20251023 20:11:43.906041 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:11:43.906217 9687 external_mini_cluster.cc:1468] Setting key 0a7da42107764bc96c6a390773e93f4e
Oct 23 20:11:43 dist-test-slave-x62p krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250303, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
I20251023 20:11:43.916265 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:43.906239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:57645 (local address 127.9.117.254:46293)
1023 20:11:43.906441 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:43.906444 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:43.906870 (+ 426us) server_negotiation.cc:408] Connection header received
1023 20:11:43.907497 (+ 627us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:43.907498 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:43.907512 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:43.907571 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:43.908436 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.909010 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:43.909653 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.909838 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:43.913080 (+ 3242us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:43.913085 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:43.913086 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:43.913094 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:43.914588 (+ 1494us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:43.915164 (+ 576us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:43.915166 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:43.915166 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:43.915198 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:43.915490 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:43.915491 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:43.915492 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:43.915604 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:43.915676 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:43.916099 (+ 423us) server_negotiation.cc:300] Negotiation successful
1023 20:11:43.916192 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20251023 20:11:43.916396 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:43.906218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46293 (local address 127.9.117.194:57645)
1023 20:11:43.906602 (+ 384us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:43.906641 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:43.907397 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:43.907643 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:43.907647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:43.908015 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:43.908331 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:43.908335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.909104 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:43.909105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:43.909542 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:43.909545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:43.909736 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:43.910930 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:43.910965 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:43.912959 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:43.914697 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:43.914707 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:43.914721 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:43.915052 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:43.915284 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:43.915285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:43.915286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:43.915389 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:43.915679 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:43.915685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:43.915916 (+ 231us) client_negotiation.cc:769] Sending connection context
1023 20:11:43.916062 (+ 146us) client_negotiation.cc:240] Negotiation successful
1023 20:11:43.916206 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":95,"threads_started":1}
I20251023 20:11:43.916965 10071 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46293
I20251023 20:11:43.917074 10071 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:43.917265 10071 heartbeater.cc:507] Master 127.9.117.254:46293 requested a full tablet report, sending...
I20251023 20:11:43.917655 9742 ts_manager.cc:194] Registered new tserver with Master: 60e6003f1ea74877a278c62b256e4dec (127.9.117.194:33233)
I20251023 20:11:43.918103 9742 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:57645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:43.960942 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46293
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:44.046847 10078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:44.047066 10078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:44.047109 10078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:44.048604 10078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:44.048652 10078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:44.048695 10078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:11:44.050349 10078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:44.050623 10078 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:44.050822 10078 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:44.053488 10084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:44.053577 10083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:44.053735 10078 server_base.cc:1047] running on GCE node
W20251023 20:11:44.053488 10086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:44.053993 10078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:44.054209 10078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:44.055354 10078 hybrid_clock.cc:648] HybridClock initialized: now 1761250304055333 us; error 39 us; skew 500 ppm
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250304, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:44.057960 10078 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:11:44.058523 10078 webserver.cc:492] Webserver started at http://127.9.117.195:37765/ using document root <none> and password file <none>
I20251023 20:11:44.058720 10078 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:44.058769 10078 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:44.058894 10078 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:44.059899 10078 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "698f4fde4ef04e2f96e8c60ccb113541"
format_stamp: "Formatted at 2025-10-23 20:11:44 on dist-test-slave-x62p"
server_key: "db9f68748d66d4c174d007f2413f5db1"
server_key_iv: "006ddf429a3078dfc2362c34538c38ea"
server_key_version: "encryptionkey@0"
I20251023 20:11:44.060215 10078 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "698f4fde4ef04e2f96e8c60ccb113541"
format_stamp: "Formatted at 2025-10-23 20:11:44 on dist-test-slave-x62p"
server_key: "db9f68748d66d4c174d007f2413f5db1"
server_key_iv: "006ddf429a3078dfc2362c34538c38ea"
server_key_version: "encryptionkey@0"
I20251023 20:11:44.061569 10078 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251023 20:11:44.062255 10093 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:44.062436 10078 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:11:44.062511 10078 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "698f4fde4ef04e2f96e8c60ccb113541"
format_stamp: "Formatted at 2025-10-23 20:11:44 on dist-test-slave-x62p"
server_key: "db9f68748d66d4c174d007f2413f5db1"
server_key_iv: "006ddf429a3078dfc2362c34538c38ea"
server_key_version: "encryptionkey@0"
I20251023 20:11:44.062582 10078 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:44.085471 10078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:44.087539 10078 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:44.087675 10078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:44.087957 10078 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:11:44.088323 10078 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:44.088359 10078 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:44.088397 10078 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:44.088423 10078 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:44.094453 10078 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:40945
I20251023 20:11:44.094547 10206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:40945 every 8 connection(s)
I20251023 20:11:44.094818 10078 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:11:44.095659 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 10078
I20251023 20:11:44.095734 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:11:44.095896 9687 external_mini_cluster.cc:1468] Setting key f1b5425ea74cfeeb5efa2dd86b15779b
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250304, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:44.103606 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:44.095604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:44413 (local address 127.9.117.254:46293)
1023 20:11:44.095681 (+ 77us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:44.095684 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:44.095830 (+ 146us) server_negotiation.cc:408] Connection header received
1023 20:11:44.096459 (+ 629us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:44.096460 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:44.096473 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:44.096510 (+ 37us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:44.097476 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:44.097969 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:44.098494 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:44.098610 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:44.100979 (+ 2369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:44.100984 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:44.100984 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:44.100991 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:44.102447 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:44.102810 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:44.102811 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:44.102811 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:44.102838 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:44.103013 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:44.103014 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:44.103014 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:44.103108 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:44.103166 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:44.103414 (+ 248us) server_negotiation.cc:300] Negotiation successful
1023 20:11:44.103506 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251023 20:11:44.103744 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:44.095543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46293 (local address 127.9.117.195:44413)
1023 20:11:44.095706 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:44.095746 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:44.096368 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:44.096623 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:44.096628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:44.097080 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:44.097391 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:44.097395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:44.098051 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:44.098052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:44.098434 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:44.098435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:44.098558 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:44.099444 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:44.099459 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:44.100894 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:44.102527 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:44.102532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:44.102535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:44.102740 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:44.102899 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:44.102900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:44.102900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:44.102958 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:44.103162 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:44.103167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:44.103294 (+ 127us) client_negotiation.cc:769] Sending connection context
1023 20:11:44.103391 (+ 97us) client_negotiation.cc:240] Negotiation successful
1023 20:11:44.103528 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20251023 20:11:44.104205 10207 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46293
I20251023 20:11:44.104303 10207 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:44.104473 10207 heartbeater.cc:507] Master 127.9.117.254:46293 requested a full tablet report, sending...
I20251023 20:11:44.104867 9742 ts_manager.cc:194] Registered new tserver with Master: 698f4fde4ef04e2f96e8c60ccb113541 (127.9.117.195:40945)
I20251023 20:11:44.105222 9742 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:44413
I20251023 20:11:44.107895 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:11:44.130503 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:44.122937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.9.117.193:36823)
1023 20:11:44.123112 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:44.123121 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:44.123132 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:11:44.123164 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:44.123166 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:44.123179 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:44.123244 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:44.123964 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:44.124691 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:44.125302 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:44.125544 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:44.127539 (+ 1995us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:44.127548 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:44.127548 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:44.127555 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:44.129380 (+ 1825us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:44.129747 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:44.129750 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:44.129751 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:44.129792 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:44.129994 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:44.129995 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:44.129995 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:44.130107 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:44.130228 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:44.130311 (+ 83us) server_negotiation.cc:300] Negotiation successful
1023 20:11:44.130393 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
W20251023 20:11:44.130875 9849 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43846
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:11:44.156319 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:44.148999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.9.117.193:36823)
1023 20:11:44.149102 (+ 103us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:44.149104 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:44.149260 (+ 156us) server_negotiation.cc:408] Connection header received
1023 20:11:44.149424 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:44.149426 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:44.149443 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:44.149514 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:44.150476 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:44.151028 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:44.151708 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:44.151940 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:44.154100 (+ 2160us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:44.154104 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:44.154104 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:44.154111 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:44.155439 (+ 1328us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:44.155765 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:44.155766 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:44.155766 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:44.155792 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:44.155911 (+ 119us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:44.155912 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:44.155912 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:44.155985 (+ 73us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:44.156059 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:44.156148 (+ 89us) server_negotiation.cc:300] Negotiation successful
1023 20:11:44.156219 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251023 20:11:44.157033 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 9806
I20251023 20:11:44.162503 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 9942
I20251023 20:11:44.168567 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 10078
I20251023 20:11:44.173241 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 9710
2025-10-23T20:11:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4055 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[10221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[10221](info): set up 2 sockets
Oct 23 20:11:44 dist-test-slave-x62p krb5kdc[10221](info): commencing operation
krb5kdc: starting...
W20251023 20:11:46.172673 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:11:46 dist-test-slave-x62p krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:11:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:46.313799 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:41109
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:41109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:46.397241 10237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:46.397486 10237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:46.397527 10237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:46.399170 10237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:11:46.399235 10237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:46.399263 10237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:46.399286 10237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:11:46.399307 10237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:11:46.401198 10237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:41109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:41109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:46.401499 10237 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:46.401705 10237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:46.404448 10242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:46.404522 10245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:46.404448 10243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:46.404719 10237 server_base.cc:1047] running on GCE node
I20251023 20:11:46.404961 10237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:46.405206 10237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:46.406355 10237 hybrid_clock.cc:648] HybridClock initialized: now 1761250306406337 us; error 30 us; skew 500 ppm
Oct 23 20:11:46 dist-test-slave-x62p krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:46.408951 10237 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:11:46.409513 10237 webserver.cc:492] Webserver started at http://127.9.117.254:45457/ using document root <none> and password file <none>
I20251023 20:11:46.409694 10237 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:46.409730 10237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:46.409839 10237 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:46.410811 10237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "012718c1ff2642c3ab12ee7d9c07ef44"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "bcbff964a69adc111216fb71f795eb2c"
server_key_iv: "7da390e9a1ca269958547c03321b906a"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.411116 10237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "012718c1ff2642c3ab12ee7d9c07ef44"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "bcbff964a69adc111216fb71f795eb2c"
server_key_iv: "7da390e9a1ca269958547c03321b906a"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.412359 10237 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20251023 20:11:46.413092 10252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:46.413251 10237 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:46.413324 10237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "012718c1ff2642c3ab12ee7d9c07ef44"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "bcbff964a69adc111216fb71f795eb2c"
server_key_iv: "7da390e9a1ca269958547c03321b906a"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.413384 10237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:46.430346 10237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:46.442209 10237 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:46.442350 10237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:46.446623 10237 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:41109
I20251023 20:11:46.446656 10304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:41109 every 8 connection(s)
I20251023 20:11:46.447000 10237 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:11:46.447578 10305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:46.449083 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 10237
I20251023 20:11:46.449173 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:11:46.449343 9687 external_mini_cluster.cc:1468] Setting key 9695d34e8cb0f63b383cd15bddbfc106
I20251023 20:11:46.451180 10305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44: Bootstrap starting.
I20251023 20:11:46.451817 10305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:46.452072 10305 log.cc:826] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:46.452701 10305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44: No bootstrap required, opened a new log
I20251023 20:11:46.453943 10305 raft_consensus.cc:359] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012718c1ff2642c3ab12ee7d9c07ef44" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41109 } }
I20251023 20:11:46.454068 10305 raft_consensus.cc:385] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:46.454103 10305 raft_consensus.cc:740] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012718c1ff2642c3ab12ee7d9c07ef44, State: Initialized, Role: FOLLOWER
I20251023 20:11:46.454205 10305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012718c1ff2642c3ab12ee7d9c07ef44" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41109 } }
I20251023 20:11:46.454280 10305 raft_consensus.cc:399] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:11:46.454324 10305 raft_consensus.cc:493] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:11:46.454372 10305 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 0 FOLLOWER]: Advancing to term 1
Oct 23 20:11:46 dist-test-slave-x62p krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:46.455029 10305 raft_consensus.cc:515] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012718c1ff2642c3ab12ee7d9c07ef44" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41109 } }
I20251023 20:11:46.455137 10305 leader_election.cc:304] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012718c1ff2642c3ab12ee7d9c07ef44; no voters:
I20251023 20:11:46.455313 10305 leader_election.cc:290] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:11:46.455364 10310 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:46.455539 10310 raft_consensus.cc:697] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [term 1 LEADER]: Becoming Leader. State: Replica: 012718c1ff2642c3ab12ee7d9c07ef44, State: Running, Role: LEADER
I20251023 20:11:46.455576 10305 sys_catalog.cc:565] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:11:46.455655 10310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012718c1ff2642c3ab12ee7d9c07ef44" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41109 } }
I20251023 20:11:46.456002 10310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012718c1ff2642c3ab12ee7d9c07ef44. Latest consensus state: current_term: 1 leader_uuid: "012718c1ff2642c3ab12ee7d9c07ef44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012718c1ff2642c3ab12ee7d9c07ef44" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41109 } } }
I20251023 20:11:46.456140 10310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [sys.catalog]: This master's current role is: LEADER
I20251023 20:11:46.456425 10312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "012718c1ff2642c3ab12ee7d9c07ef44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012718c1ff2642c3ab12ee7d9c07ef44" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41109 } } }
I20251023 20:11:46.456573 10317 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:11:46.456593 10312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44 [sys.catalog]: This master's current role is: LEADER
I20251023 20:11:46.457190 10317 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:11:46.459270 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:46.449857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.9.117.254:41109)
1023 20:11:46.450004 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:46.450013 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:46.450032 (+ 19us) server_negotiation.cc:408] Connection header received
1023 20:11:46.450344 (+ 312us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:46.450349 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:46.450388 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:46.450494 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:46.451452 (+ 958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.452275 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:46.452909 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.453174 (+ 265us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:46.455203 (+ 2029us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:46.455210 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:46.455211 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:46.455235 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:46.457942 (+ 2707us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:46.458286 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:46.458289 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:46.458289 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:46.458353 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:46.458488 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:46.458488 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:46.458489 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:46.458740 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:46.458892 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:46.458975 (+ 83us) server_negotiation.cc:300] Negotiation successful
1023 20:11:46.459099 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20251023 20:11:46.461804 10317 catalog_manager.cc:1357] Generated new cluster ID: 0f9028415d984131bec590aec4e22744
I20251023 20:11:46.461856 10317 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:11:46.476243 10317 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:11:46.476895 10317 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:11:46.481734 10317 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 012718c1ff2642c3ab12ee7d9c07ef44: Generated new TSK 0
I20251023 20:11:46.481948 10317 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:46.533721 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:41109
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:46.616420 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:46.616677 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:46.616743 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:46.618383 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:46.618438 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:46.618481 10333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:11:46.620118 10333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:41109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:46.620393 10333 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:46.620594 10333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:46.623191 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:46.623240 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:46.623199 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:46.623467 10333 server_base.cc:1047] running on GCE node
I20251023 20:11:46.623633 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:46.623816 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:46.624945 10333 hybrid_clock.cc:648] HybridClock initialized: now 1761250306624934 us; error 29 us; skew 500 ppm
Oct 23 20:11:46 dist-test-slave-x62p krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:46.627326 10333 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:11:46.627898 10333 webserver.cc:492] Webserver started at http://127.9.117.193:34075/ using document root <none> and password file <none>
I20251023 20:11:46.628089 10333 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:46.628129 10333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:46.628247 10333 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:46.629282 10333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "38a9e83db6a4406db5a9e674cb2151df"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "b99aa12cc4d162f9b4b0d7820b41e873"
server_key_iv: "cc3ec8594f85b282ca1c0a930fba3b3b"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.629613 10333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "38a9e83db6a4406db5a9e674cb2151df"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "b99aa12cc4d162f9b4b0d7820b41e873"
server_key_iv: "cc3ec8594f85b282ca1c0a930fba3b3b"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.630851 10333 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:11:46.631479 10348 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:46.631675 10333 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:11:46.631749 10333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "38a9e83db6a4406db5a9e674cb2151df"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "b99aa12cc4d162f9b4b0d7820b41e873"
server_key_iv: "cc3ec8594f85b282ca1c0a930fba3b3b"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.631814 10333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:46.640651 10333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:46.642489 10333 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:46.642630 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:46.642853 10333 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:11:46.643170 10333 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:46.643203 10333 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:46.643239 10333 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:46.643265 10333 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:46.649135 10333 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:35867
I20251023 20:11:46.649206 10461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:35867 every 8 connection(s)
I20251023 20:11:46.649487 10333 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
Oct 23 20:11:46 dist-test-slave-x62p krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:46.658421 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 10333
I20251023 20:11:46.658535 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:11:46.658705 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:46.650220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:41497 (local address 127.9.117.254:41109)
1023 20:11:46.650268 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:46.650271 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:46.650440 (+ 169us) server_negotiation.cc:408] Connection header received
1023 20:11:46.650986 (+ 546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:46.650988 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:46.651001 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:46.651061 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:46.651858 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.652355 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:46.653581 (+ 1226us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.653765 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:46.655803 (+ 2038us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:46.655810 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:46.655810 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:46.655822 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:46.657367 (+ 1545us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:46.657785 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:46.657786 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:46.657786 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:46.657813 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:46.657972 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:46.657973 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:46.657973 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:46.658072 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:46.658121 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:46.658516 (+ 395us) server_negotiation.cc:300] Negotiation successful
1023 20:11:46.658635 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251023 20:11:46.658736 9687 external_mini_cluster.cc:1468] Setting key 93b08b06eefb48d39e9afda8216bc259
I20251023 20:11:46.658892 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:46.650153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:41109 (local address 127.9.117.193:41497)
1023 20:11:46.650329 (+ 176us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:46.650361 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:46.650889 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:46.651142 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:46.651146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:46.651468 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:46.651764 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:46.651767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.652441 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:46.652442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:46.653490 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:46.653493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.653647 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:46.654194 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:46.654212 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:46.655692 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:46.657450 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:46.657456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:46.657479 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:46.657694 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:46.657871 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:46.657872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:46.657872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:46.657929 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:46.658128 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:46.658133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:46.658320 (+ 187us) client_negotiation.cc:769] Sending connection context
1023 20:11:46.658489 (+ 169us) client_negotiation.cc:240] Negotiation successful
1023 20:11:46.658673 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20251023 20:11:46.659467 10462 heartbeater.cc:344] Connected to a master server at 127.9.117.254:41109
I20251023 20:11:46.659598 10462 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:46.659806 10462 heartbeater.cc:507] Master 127.9.117.254:41109 requested a full tablet report, sending...
I20251023 20:11:46.660312 10268 ts_manager.cc:194] Registered new tserver with Master: 38a9e83db6a4406db5a9e674cb2151df (127.9.117.193:35867)
I20251023 20:11:46.661055 10268 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:41497
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:46.704672 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:41109
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:46.787048 10469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:46.787233 10469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:46.787258 10469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:46.788650 10469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:46.788697 10469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:46.788722 10469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:11:46.790342 10469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:41109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:46.790619 10469 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:46.790827 10469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:46.793586 10475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:46.793597 10477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:46.793608 10474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:46.794027 10469 server_base.cc:1047] running on GCE node
I20251023 20:11:46.794196 10469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:46.794397 10469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:46.795533 10469 hybrid_clock.cc:648] HybridClock initialized: now 1761250306795493 us; error 61 us; skew 500 ppm
Oct 23 20:11:46 dist-test-slave-x62p krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:46.797959 10469 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:11:46.798522 10469 webserver.cc:492] Webserver started at http://127.9.117.194:41825/ using document root <none> and password file <none>
I20251023 20:11:46.798697 10469 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:46.798731 10469 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:46.798847 10469 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:46.799841 10469 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "c46dba1f1ffa483a823d3a51ac5724e1"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "fd0c4c077c1679804cb3743fd543518e"
server_key_iv: "223c3d6d892f694ea29756fec378cced"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.800163 10469 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "c46dba1f1ffa483a823d3a51ac5724e1"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "fd0c4c077c1679804cb3743fd543518e"
server_key_iv: "223c3d6d892f694ea29756fec378cced"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.801493 10469 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:11:46.802268 10484 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:46.802484 10469 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:11:46.802567 10469 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "c46dba1f1ffa483a823d3a51ac5724e1"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "fd0c4c077c1679804cb3743fd543518e"
server_key_iv: "223c3d6d892f694ea29756fec378cced"
server_key_version: "encryptionkey@0"
I20251023 20:11:46.802624 10469 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:46.831554 10469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:46.833603 10469 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:46.833734 10469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:46.833948 10469 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:11:46.834235 10469 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:46.834265 10469 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:46.834287 10469 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:46.834306 10469 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:46.840437 10469 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:42879
I20251023 20:11:46.840497 10597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:42879 every 8 connection(s)
I20251023 20:11:46.840849 10469 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:11:46 dist-test-slave-x62p krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:46.849681 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 10469
I20251023 20:11:46.849807 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:11:46.849813 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:46.841622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:34423 (local address 127.9.117.254:41109)
1023 20:11:46.841680 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:46.841682 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:46.841835 (+ 153us) server_negotiation.cc:408] Connection header received
1023 20:11:46.842420 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:46.842421 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:46.842431 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:46.842477 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:46.843299 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.843829 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:46.844937 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.845058 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:46.846982 (+ 1924us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:46.846987 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:46.846988 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:46.846994 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:46.848437 (+ 1443us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:46.848935 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:46.848936 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:46.848937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:46.848963 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:46.849169 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:46.849171 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:46.849171 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:46.849273 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:46.849336 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:46.849608 (+ 272us) server_negotiation.cc:300] Negotiation successful
1023 20:11:46.849708 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:11:46.849861 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:46.841553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:41109 (local address 127.9.117.194:34423)
1023 20:11:46.841718 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:46.841752 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:46.842310 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:46.842556 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:46.842560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:46.842908 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:46.843202 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:46.843205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.843915 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:46.843916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:46.844850 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:46.844853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:46.844968 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:46.845515 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:46.845531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:46.846897 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:46.848532 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:46.848538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:46.848541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:46.848805 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:46.849030 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:46.849031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:46.849031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:46.849095 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:46.849329 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:46.849331 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:46.849452 (+ 121us) client_negotiation.cc:769] Sending connection context
1023 20:11:46.849552 (+ 100us) client_negotiation.cc:240] Negotiation successful
1023 20:11:46.849672 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20251023 20:11:46.849984 9687 external_mini_cluster.cc:1468] Setting key d726662d563c53aa66995e15ff697ba4
I20251023 20:11:46.850476 10598 heartbeater.cc:344] Connected to a master server at 127.9.117.254:41109
I20251023 20:11:46.850597 10598 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:46.850831 10598 heartbeater.cc:507] Master 127.9.117.254:41109 requested a full tablet report, sending...
I20251023 20:11:46.851223 10269 ts_manager.cc:194] Registered new tserver with Master: c46dba1f1ffa483a823d3a51ac5724e1 (127.9.117.194:42879)
I20251023 20:11:46.851837 10269 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:34423
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:46.899659 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:41109
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:46.987257 10605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:46.987478 10605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:46.987525 10605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:46.989059 10605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:46.989110 10605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:46.989151 10605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:11:46.990921 10605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:41109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:46.991204 10605 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:46.991400 10605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:46.994421 10610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:46.994444 10605 server_base.cc:1047] running on GCE node
W20251023 20:11:46.994593 10611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:46.994804 10613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:46.995045 10605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:46.995261 10605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:46.996423 10605 hybrid_clock.cc:648] HybridClock initialized: now 1761250306996409 us; error 29 us; skew 500 ppm
Oct 23 20:11:46 dist-test-slave-x62p krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:46.999051 10605 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:11:46.999671 10605 webserver.cc:492] Webserver started at http://127.9.117.195:45887/ using document root <none> and password file <none>
I20251023 20:11:46.999873 10605 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:46.999917 10605 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:47.000037 10605 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:47.001116 10605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "3e4331b9a3ee4a6b80ea454315c6ac80"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "7b1dfe49ac059c7f0fd0b15e59ce4933"
server_key_iv: "ac8dfaf101ea09c416daaa672342e496"
server_key_version: "encryptionkey@0"
I20251023 20:11:47.001441 10605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "3e4331b9a3ee4a6b80ea454315c6ac80"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "7b1dfe49ac059c7f0fd0b15e59ce4933"
server_key_iv: "ac8dfaf101ea09c416daaa672342e496"
server_key_version: "encryptionkey@0"
I20251023 20:11:47.002739 10605 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:11:47.003561 10620 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:47.003798 10605 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:11:47.003891 10605 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "3e4331b9a3ee4a6b80ea454315c6ac80"
format_stamp: "Formatted at 2025-10-23 20:11:46 on dist-test-slave-x62p"
server_key: "7b1dfe49ac059c7f0fd0b15e59ce4933"
server_key_iv: "ac8dfaf101ea09c416daaa672342e496"
server_key_version: "encryptionkey@0"
I20251023 20:11:47.003961 10605 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:47.031806 10605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:47.034081 10605 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:47.034238 10605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:47.034477 10605 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:11:47.034826 10605 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:47.034937 10605 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:47.034966 10605 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:47.034983 10605 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:47.042197 10605 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:38775
I20251023 20:11:47.042255 10733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:38775 every 8 connection(s)
I20251023 20:11:47.042634 10605 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:11:47.045056 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 10605
I20251023 20:11:47.045140 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:11:47.045279 9687 external_mini_cluster.cc:1468] Setting key 5137d463862fb65525fa9b7473e46319
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:47.051841 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.043387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:34823 (local address 127.9.117.254:41109)
1023 20:11:47.043437 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.043441 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.043634 (+ 193us) server_negotiation.cc:408] Connection header received
1023 20:11:47.044179 (+ 545us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.044181 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.044194 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.044258 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:47.045114 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.045834 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.046929 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.047099 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.049191 (+ 2092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:47.049195 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:47.049195 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:47.049202 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:47.050601 (+ 1399us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.051001 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.051002 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.051002 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.051029 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.051208 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.051209 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.051209 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.051311 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:47.051373 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.051655 (+ 282us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.051747 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:11:47.052016 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.043334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:41109 (local address 127.9.117.195:34823)
1023 20:11:47.043512 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:47.043548 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:47.044082 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:47.044331 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:47.044335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:47.044668 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:47.045020 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.045023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.045961 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.045962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:47.046846 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.046848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.046997 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.047577 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:47.047594 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:47.049095 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:47.050694 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:47.050701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:47.050712 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:47.050930 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:47.051084 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:47.051085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:47.051085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:47.051152 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:47.051388 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:47.051393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:47.051527 (+ 134us) client_negotiation.cc:769] Sending connection context
1023 20:11:47.051629 (+ 102us) client_negotiation.cc:240] Negotiation successful
1023 20:11:47.051789 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20251023 20:11:47.052498 10734 heartbeater.cc:344] Connected to a master server at 127.9.117.254:41109
I20251023 20:11:47.052601 10734 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:47.052857 10734 heartbeater.cc:507] Master 127.9.117.254:41109 requested a full tablet report, sending...
I20251023 20:11:47.053264 10269 ts_manager.cc:194] Registered new tserver with Master: 3e4331b9a3ee4a6b80ea454315c6ac80 (127.9.117.195:38775)
I20251023 20:11:47.053671 10269 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:34823
I20251023 20:11:47.057307 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:11:47.064231 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.058842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.9.117.254:41109)
1023 20:11:47.058894 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.058896 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.058943 (+ 47us) server_negotiation.cc:408] Connection header received
1023 20:11:47.058988 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.058989 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.058997 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.059052 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:47.059747 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.060223 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.060904 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.061023 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.062026 (+ 1003us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:47.062030 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:47.062030 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:47.062036 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:47.063318 (+ 1282us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.063637 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.063638 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.063639 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.063664 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.063811 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.063812 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.063812 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.063937 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:47.064018 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.064094 (+ 76us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.064162 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:11:47.066429 10269 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:11:47.067759 10269 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:47.073585 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.070272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42879 (local address 127.0.0.1:59822)
1023 20:11:47.070510 (+ 238us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:47.070519 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:47.070555 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:47.070812 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:47.070815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:47.070817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:47.071144 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.071147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.072567 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.072569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:47.073277 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.073279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.073344 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.073389 (+ 45us) client_negotiation.cc:769] Sending connection context
1023 20:11:47.073440 (+ 51us) client_negotiation.cc:240] Negotiation successful
1023 20:11:47.073495 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
I20251023 20:11:47.074069 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.070478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.9.117.194:42879)
1023 20:11:47.070629 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.070636 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.070646 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:11:47.070682 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.070685 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.070702 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.070787 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:47.071248 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.072462 (+ 1214us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.073381 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.073912 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.073957 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.073981 (+ 24us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.074011 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20251023 20:11:47.074388 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.070072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:38775 (local address 127.0.0.1:59942)
1023 20:11:47.070288 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:47.070316 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:47.070408 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:47.071350 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:47.071351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:47.071354 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:47.071510 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.071511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.072389 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.072390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:47.073296 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.073299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.073362 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.073392 (+ 30us) client_negotiation.cc:769] Sending connection context
1023 20:11:47.073435 (+ 43us) client_negotiation.cc:240] Negotiation successful
1023 20:11:47.073474 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
I20251023 20:11:47.074697 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.070072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:35867 (local address 127.0.0.1:34282)
1023 20:11:47.070388 (+ 316us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:47.070398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:47.070452 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:47.070812 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:47.070815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:47.070820 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:47.071144 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.071147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.072540 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.072541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:47.073569 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.073571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.073631 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.073633 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:11:47.074598 (+ 965us) client_negotiation.cc:240] Negotiation successful
1023 20:11:47.074644 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20251023 20:11:47.074659 10532 tablet_service.cc:1505] Processing CreateTablet for tablet 54d7ee41fae64631b1640a75b36d795c (DEFAULT_TABLE table=test-table [id=f10af873be8a4037aefe47a10e49d523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:47.074772 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.070171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.9.117.195:38775)
1023 20:11:47.071037 (+ 866us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.071047 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.071058 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:11:47.071168 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.071171 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.071190 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.071269 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:47.071588 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.072295 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.073559 (+ 1264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.074104 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.074592 (+ 488us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.074625 (+ 33us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.074681 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":106,"threads_started":1}
I20251023 20:11:47.074956 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.070264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.9.117.193:35867)
1023 20:11:47.070613 (+ 349us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.070623 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.070637 (+ 14us) server_negotiation.cc:408] Connection header received
1023 20:11:47.070683 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.070685 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.070702 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.070790 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:47.071248 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.072455 (+ 1207us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.073927 (+ 1472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.074458 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.074531 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.074817 (+ 286us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.074882 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20251023 20:11:47.075004 10532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d7ee41fae64631b1640a75b36d795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:47.075284 10668 tablet_service.cc:1505] Processing CreateTablet for tablet 54d7ee41fae64631b1640a75b36d795c (DEFAULT_TABLE table=test-table [id=f10af873be8a4037aefe47a10e49d523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:47.075559 10668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d7ee41fae64631b1640a75b36d795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:47.077517 10751 tablet_bootstrap.cc:492] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1: Bootstrap starting.
I20251023 20:11:47.077952 10752 tablet_bootstrap.cc:492] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80: Bootstrap starting.
I20251023 20:11:47.078145 10751 tablet_bootstrap.cc:654] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:47.078354 10396 tablet_service.cc:1505] Processing CreateTablet for tablet 54d7ee41fae64631b1640a75b36d795c (DEFAULT_TABLE table=test-table [id=f10af873be8a4037aefe47a10e49d523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:47.078467 10751 log.cc:826] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:47.078591 10396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d7ee41fae64631b1640a75b36d795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:47.078842 10752 tablet_bootstrap.cc:654] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:47.079166 10752 log.cc:826] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:47.079229 10751 tablet_bootstrap.cc:492] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1: No bootstrap required, opened a new log
I20251023 20:11:47.079308 10751 ts_tablet_manager.cc:1403] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:11:47.079879 10752 tablet_bootstrap.cc:492] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80: No bootstrap required, opened a new log
I20251023 20:11:47.079943 10752 ts_tablet_manager.cc:1403] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:11:47.081166 10752 raft_consensus.cc:359] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.081288 10752 raft_consensus.cc:385] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:47.081321 10752 raft_consensus.cc:740] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4331b9a3ee4a6b80ea454315c6ac80, State: Initialized, Role: FOLLOWER
I20251023 20:11:47.081239 10751 raft_consensus.cc:359] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.081374 10751 raft_consensus.cc:385] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:47.081405 10751 raft_consensus.cc:740] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46dba1f1ffa483a823d3a51ac5724e1, State: Initialized, Role: FOLLOWER
I20251023 20:11:47.081630 10751 consensus_queue.cc:260] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.081444 10752 consensus_queue.cc:260] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.081940 10598 heartbeater.cc:499] Master 127.9.117.254:41109 was elected leader, sending a full tablet report...
I20251023 20:11:47.081949 10752 ts_tablet_manager.cc:1434] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:11:47.081956 10734 heartbeater.cc:499] Master 127.9.117.254:41109 was elected leader, sending a full tablet report...
I20251023 20:11:47.081928 10751 ts_tablet_manager.cc:1434] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:11:47.082155 10755 tablet_bootstrap.cc:492] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df: Bootstrap starting.
I20251023 20:11:47.082799 10755 tablet_bootstrap.cc:654] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:47.083140 10755 log.cc:826] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:47.083793 10755 tablet_bootstrap.cc:492] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df: No bootstrap required, opened a new log
I20251023 20:11:47.083844 10755 ts_tablet_manager.cc:1403] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:11:47.085137 10755 raft_consensus.cc:359] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.085254 10755 raft_consensus.cc:385] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:47.085274 10755 raft_consensus.cc:740] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a9e83db6a4406db5a9e674cb2151df, State: Initialized, Role: FOLLOWER
I20251023 20:11:47.085371 10755 consensus_queue.cc:260] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.085610 10755 ts_tablet_manager.cc:1434] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:11:47.085644 10462 heartbeater.cc:499] Master 127.9.117.254:41109 was elected leader, sending a full tablet report...
W20251023 20:11:47.091398 10599 tablet.cc:2378] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:11:47.139846 10757 raft_consensus.cc:493] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:47.139981 10757 raft_consensus.cc:515] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.140304 10757 leader_election.cc:290] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38a9e83db6a4406db5a9e674cb2151df (127.9.117.193:35867), c46dba1f1ffa483a823d3a51ac5724e1 (127.9.117.194:42879)
I20251023 20:11:47.142894 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.140499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:35867 (local address 127.9.117.195:52359)
1023 20:11:47.140592 (+ 93us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:47.140608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:47.140701 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:47.140931 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:47.140932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:47.140936 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:47.141198 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.141201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.141929 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.141931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:47.142631 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.142634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.142711 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.142717 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:11:47.142753 (+ 36us) client_negotiation.cc:240] Negotiation successful
1023 20:11:47.142817 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20251023 20:11:47.143077 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.140644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42879 (local address 127.9.117.195:37031)
1023 20:11:47.140846 (+ 202us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:47.140857 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:47.140932 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:47.141080 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:47.141080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:47.141082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:47.141347 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.141348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.142107 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.142108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:47.142769 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:47.142772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.142890 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.142894 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:11:47.142928 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:11:47.142966 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":125,"threads_started":1}
I20251023 20:11:47.143404 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.140621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:52359 (local address 127.9.117.193:35867)
1023 20:11:47.140677 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.140680 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.140693 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:11:47.140822 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.140824 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.140836 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.140911 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:47.141280 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.141828 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.142743 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.143296 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.143305 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.143327 (+ 22us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.143356 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:11:47.143524 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.140782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:37031 (local address 127.9.117.194:42879)
1023 20:11:47.140833 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.140835 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.140922 (+ 87us) server_negotiation.cc:408] Connection header received
1023 20:11:47.140990 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.140992 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.141005 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.141084 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:47.141460 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.142013 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.142909 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.143397 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.143404 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.143426 (+ 22us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.143452 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:11:47.143747 10416 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d7ee41fae64631b1640a75b36d795c" candidate_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a9e83db6a4406db5a9e674cb2151df" is_pre_election: true
I20251023 20:11:47.143839 10552 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d7ee41fae64631b1640a75b36d795c" candidate_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" is_pre_election: true
I20251023 20:11:47.143937 10416 raft_consensus.cc:2468] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e4331b9a3ee4a6b80ea454315c6ac80 in term 0.
I20251023 20:11:47.143968 10552 raft_consensus.cc:2468] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e4331b9a3ee4a6b80ea454315c6ac80 in term 0.
I20251023 20:11:47.144183 10622 leader_election.cc:304] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a9e83db6a4406db5a9e674cb2151df, 3e4331b9a3ee4a6b80ea454315c6ac80; no voters:
I20251023 20:11:47.144315 10757 raft_consensus.cc:2804] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:47.144379 10757 raft_consensus.cc:493] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:47.144412 10757 raft_consensus.cc:3060] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:47.145150 10757 raft_consensus.cc:515] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.145272 10757 leader_election.cc:290] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [CANDIDATE]: Term 1 election: Requested vote from peers 38a9e83db6a4406db5a9e674cb2151df (127.9.117.193:35867), c46dba1f1ffa483a823d3a51ac5724e1 (127.9.117.194:42879)
I20251023 20:11:47.145476 10416 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d7ee41fae64631b1640a75b36d795c" candidate_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a9e83db6a4406db5a9e674cb2151df"
I20251023 20:11:47.145483 10552 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54d7ee41fae64631b1640a75b36d795c" candidate_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46dba1f1ffa483a823d3a51ac5724e1"
I20251023 20:11:47.145551 10552 raft_consensus.cc:3060] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:47.145555 10416 raft_consensus.cc:3060] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:47.146410 10552 raft_consensus.cc:2468] T 54d7ee41fae64631b1640a75b36d795c P c46dba1f1ffa483a823d3a51ac5724e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e4331b9a3ee4a6b80ea454315c6ac80 in term 1.
I20251023 20:11:47.146425 10416 raft_consensus.cc:2468] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e4331b9a3ee4a6b80ea454315c6ac80 in term 1.
I20251023 20:11:47.146590 10622 leader_election.cc:304] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a9e83db6a4406db5a9e674cb2151df, 3e4331b9a3ee4a6b80ea454315c6ac80; no voters:
I20251023 20:11:47.146683 10757 raft_consensus.cc:2804] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:47.146840 10757 raft_consensus.cc:697] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [term 1 LEADER]: Becoming Leader. State: Replica: 3e4331b9a3ee4a6b80ea454315c6ac80, State: Running, Role: LEADER
I20251023 20:11:47.146965 10757 consensus_queue.cc:237] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } }
I20251023 20:11:47.147770 10269 catalog_manager.cc:5649] T 54d7ee41fae64631b1640a75b36d795c P 3e4331b9a3ee4a6b80ea454315c6ac80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e4331b9a3ee4a6b80ea454315c6ac80 (127.9.117.195). New cstate: current_term: 1 leader_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a9e83db6a4406db5a9e674cb2151df" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c46dba1f1ffa483a823d3a51ac5724e1" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e4331b9a3ee4a6b80ea454315c6ac80" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 38775 } health_report { overall_health: HEALTHY } } }
W20251023 20:11:47.150205 10463 tablet.cc:2378] T 54d7ee41fae64631b1640a75b36d795c P 38a9e83db6a4406db5a9e674cb2151df: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:11:47.173434 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.165908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.9.117.193:35867)
1023 20:11:47.165972 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.165974 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.166074 (+ 100us) server_negotiation.cc:408] Connection header received
1023 20:11:47.166131 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.166132 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.166143 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.166193 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:47.166956 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.167473 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.168093 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.168233 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.170607 (+ 2374us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:47.170626 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:47.170627 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:47.170637 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:47.172470 (+ 1833us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.172877 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.172880 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.172881 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.172923 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.173034 (+ 111us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.173035 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.173035 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.173136 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:47.173252 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.173281 (+ 29us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.173360 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:11:47.195297 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.188543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.9.117.193:35867)
1023 20:11:47.188608 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.188610 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.188691 (+ 81us) server_negotiation.cc:408] Connection header received
1023 20:11:47.188816 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.188818 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.188827 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.188880 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:47.189575 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.190047 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.190598 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.190798 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.192874 (+ 2076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:47.192879 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:47.192880 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:47.192889 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:47.194182 (+ 1293us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.194569 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.194571 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.194571 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.194605 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.194819 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.194820 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.194820 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.194900 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:47.194978 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.195149 (+ 171us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.195224 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:11:47.218057 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:47.211062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.9.117.193:35867)
1023 20:11:47.211118 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:47.211120 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:47.211173 (+ 53us) server_negotiation.cc:408] Connection header received
1023 20:11:47.211228 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:47.211228 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:47.211237 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:47.211298 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:47.212004 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.212514 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:47.213212 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:47.213352 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:47.215471 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:47.215476 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:47.215477 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:47.215491 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:47.217052 (+ 1561us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.217384 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.217385 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.217385 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.217410 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:47.217552 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:47.217552 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:47.217553 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:47.217639 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:47.217729 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:47.217855 (+ 126us) server_negotiation.cc:300] Negotiation successful
1023 20:11:47.217948 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20251023 20:11:47.218597 10376 tablet_service.cc:3012] Rejecting scan request for tablet 54d7ee41fae64631b1640a75b36d795c: Uninitialized: clean time has not yet been initialized
I20251023 20:11:47.219507 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 10333
I20251023 20:11:47.224799 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 10469
I20251023 20:11:47.229842 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 10605
I20251023 20:11:47.234740 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 10237
2025-10-23T20:11:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3062 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10775](info): set up 2 sockets
Oct 23 20:11:47 dist-test-slave-x62p krb5kdc[10775](info): commencing operation
krb5kdc: starting...
W20251023 20:11:49.248872 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.992s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:11:49 dist-test-slave-x62p krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:11:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:49.433651 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40487
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:40487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:49.518580 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:49.518795 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:49.518834 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:49.520424 10791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:11:49.520488 10791 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251023 20:11:49.520510 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:49.520534 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:49.520552 10791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:11:49.520589 10791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:11:49.522410 10791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:40487
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:49.522739 10791 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:49.522943 10791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:49.525553 10799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:49.525693 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:49.525700 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:49.525888 10791 server_base.cc:1047] running on GCE node
I20251023 20:11:49.526096 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:49.526316 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:49.527456 10791 hybrid_clock.cc:648] HybridClock initialized: now 1761250309527443 us; error 26 us; skew 500 ppm
Oct 23 20:11:49 dist-test-slave-x62p krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250309, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:49.529846 10791 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:11:49.530447 10791 webserver.cc:492] Webserver started at http://127.9.117.254:44169/ using document root <none> and password file <none>
I20251023 20:11:49.530640 10791 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:49.530681 10791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:49.530800 10791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:49.531839 10791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "5797e6225286431294f39098a20543ec"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "5427d4c044a1129e0dfb94c8df7bbb4f"
server_key_iv: "ff3ed9aa90f7eedd493e3a9e8d788ded"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.532159 10791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "5797e6225286431294f39098a20543ec"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "5427d4c044a1129e0dfb94c8df7bbb4f"
server_key_iv: "ff3ed9aa90f7eedd493e3a9e8d788ded"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.533479 10791 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:11:49.534204 10806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:49.534389 10791 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:11:49.534466 10791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "5797e6225286431294f39098a20543ec"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "5427d4c044a1129e0dfb94c8df7bbb4f"
server_key_iv: "ff3ed9aa90f7eedd493e3a9e8d788ded"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.534536 10791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:49.547240 10791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:49.549377 10791 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:49.549531 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:49.554800 10791 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:40487
I20251023 20:11:49.554891 10868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:40487 every 8 connection(s)
I20251023 20:11:49.555233 10791 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:11:49.556131 10869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:49.558485 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 10791
I20251023 20:11:49.558590 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:11:49.558739 9687 external_mini_cluster.cc:1468] Setting key 7e0dfeea6e8b38b427d1bee2f5519165
I20251023 20:11:49.558808 10869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec: Bootstrap starting.
I20251023 20:11:49.559597 10869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:49.559906 10869 log.cc:826] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:49.560815 10869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec: No bootstrap required, opened a new log
I20251023 20:11:49.562038 10869 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5797e6225286431294f39098a20543ec" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40487 } }
I20251023 20:11:49.562331 10869 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:49.562376 10869 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5797e6225286431294f39098a20543ec, State: Initialized, Role: FOLLOWER
I20251023 20:11:49.562484 10869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5797e6225286431294f39098a20543ec" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40487 } }
I20251023 20:11:49.562572 10869 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:11:49.562604 10869 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:11:49.562650 10869 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:49.563597 10869 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5797e6225286431294f39098a20543ec" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40487 } }
I20251023 20:11:49.563766 10869 leader_election.cc:304] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5797e6225286431294f39098a20543ec; no voters:
I20251023 20:11:49.564008 10869 leader_election.cc:290] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:11:49.564065 10874 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:49.564222 10874 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [term 1 LEADER]: Becoming Leader. State: Replica: 5797e6225286431294f39098a20543ec, State: Running, Role: LEADER
I20251023 20:11:49.564323 10869 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:11:49.564375 10874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5797e6225286431294f39098a20543ec" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40487 } }
I20251023 20:11:49.564723 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5797e6225286431294f39098a20543ec. Latest consensus state: current_term: 1 leader_uuid: "5797e6225286431294f39098a20543ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5797e6225286431294f39098a20543ec" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40487 } } }
I20251023 20:11:49.564888 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [sys.catalog]: This master's current role is: LEADER
I20251023 20:11:49.564759 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5797e6225286431294f39098a20543ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5797e6225286431294f39098a20543ec" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40487 } } }
Oct 23 20:11:49 dist-test-slave-x62p krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:49.565232 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec [sys.catalog]: This master's current role is: LEADER
I20251023 20:11:49.565855 10880 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:11:49.566367 10880 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:11:49.568835 10880 catalog_manager.cc:1357] Generated new cluster ID: f3d82805f93b48a49dd673630ac565ee
I20251023 20:11:49.568903 10880 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:11:49.570075 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.559320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.9.117.254:40487)
1023 20:11:49.559516 (+ 196us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:49.559526 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:49.559549 (+ 23us) server_negotiation.cc:408] Connection header received
1023 20:11:49.559940 (+ 391us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:49.559949 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:49.560015 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:49.560156 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:49.561207 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.561970 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:49.563099 (+ 1129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.563390 (+ 291us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:49.565805 (+ 2415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:49.565821 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:49.565821 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:49.565856 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:49.568354 (+ 2498us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:49.568791 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:49.568795 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:49.568795 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:49.568874 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:49.569087 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:49.569089 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:49.569089 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:49.569393 (+ 304us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:49.569569 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:49.569702 (+ 133us) server_negotiation.cc:300] Negotiation successful
1023 20:11:49.569874 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20251023 20:11:49.583348 10880 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:11:49.583988 10880 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:11:49.594978 10880 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec: Generated new TSK 0
I20251023 20:11:49.595274 10880 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:49.652143 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40487
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:49.737071 10897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:49.737246 10897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:49.737272 10897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:49.738791 10897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:49.738839 10897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:49.738862 10897 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:11:49.738885 10897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:11:49.740483 10897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:40487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:49.740749 10897 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:49.740949 10897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:49.743788 10903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:49.743726 10905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:49.743904 10897 server_base.cc:1047] running on GCE node
W20251023 20:11:49.743726 10902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:49.744112 10897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:49.744308 10897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:49.745442 10897 hybrid_clock.cc:648] HybridClock initialized: now 1761250309745430 us; error 30 us; skew 500 ppm
Oct 23 20:11:49 dist-test-slave-x62p krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250309, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:49.747803 10897 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:11:49.748438 10897 webserver.cc:492] Webserver started at http://127.9.117.193:39087/ using document root <none> and password file <none>
I20251023 20:11:49.748621 10897 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:49.748656 10897 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:49.748790 10897 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:49.749787 10897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "a17482232f614f468266b744a03b19e0"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "7cb54c019c4dd8499514195631d8fbd5"
server_key_iv: "a7f74b86f7a5f4f9c300e81703dec0bc"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.750082 10897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "a17482232f614f468266b744a03b19e0"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "7cb54c019c4dd8499514195631d8fbd5"
server_key_iv: "a7f74b86f7a5f4f9c300e81703dec0bc"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.751273 10897 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:11:49.751955 10912 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:49.752140 10897 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:11:49.752209 10897 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "a17482232f614f468266b744a03b19e0"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "7cb54c019c4dd8499514195631d8fbd5"
server_key_iv: "a7f74b86f7a5f4f9c300e81703dec0bc"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.752266 10897 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:49.776944 10897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:49.779191 10897 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:49.779304 10897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:49.779837 10897 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:49.779881 10897 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:49.779906 10897 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:49.779918 10897 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 23 20:11:49 dist-test-slave-x62p krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250309, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:49.786990 10897 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:40723
I20251023 20:11:49.787089 11027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:40723 every 8 connection(s)
I20251023 20:11:49.787427 10897 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:11:49.790159 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.780327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:44723 (local address 127.9.117.254:40487)
1023 20:11:49.780390 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:49.780393 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:49.780750 (+ 357us) server_negotiation.cc:408] Connection header received
1023 20:11:49.781307 (+ 557us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:49.781309 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:49.781325 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:49.781404 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:49.782434 (+ 1030us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.783336 (+ 902us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:49.784224 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.784450 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:49.787109 (+ 2659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:49.787115 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:49.787116 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:49.787125 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:49.788859 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:49.789242 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:49.789243 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:49.789244 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:49.789274 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:49.789448 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:49.789448 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:49.789449 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:49.789570 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:49.789622 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:49.789998 (+ 376us) server_negotiation.cc:300] Negotiation successful
1023 20:11:49.790089 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251023 20:11:49.790274 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.780284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40487 (local address 127.9.117.193:44723)
1023 20:11:49.780601 (+ 317us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:49.780633 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:49.781200 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:49.781450 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:49.781453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:49.781841 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:49.782292 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:49.782297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.783434 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:49.783435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:49.784118 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:49.784121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.784283 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:49.785034 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:49.785077 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:49.786978 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:49.788949 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:49.788955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:49.788957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:49.789165 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:49.789339 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:49.789340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:49.789340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:49.789403 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:49.789725 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:49.789732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:49.789881 (+ 149us) client_negotiation.cc:769] Sending connection context
1023 20:11:49.789986 (+ 105us) client_negotiation.cc:240] Negotiation successful
1023 20:11:49.790108 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":221,"threads_started":1}
I20251023 20:11:49.790832 11028 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40487
I20251023 20:11:49.790930 11028 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:49.791136 11028 heartbeater.cc:507] Master 127.9.117.254:40487 requested a full tablet report, sending...
I20251023 20:11:49.791707 10822 ts_manager.cc:194] Registered new tserver with Master: a17482232f614f468266b744a03b19e0 (127.9.117.193:40723)
I20251023 20:11:49.792441 10822 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:44723
I20251023 20:11:49.797058 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.791890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40487 (local address 127.9.117.193:42567)
1023 20:11:49.792042 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:49.792050 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:49.792113 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:49.792245 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:49.792245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:49.792392 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:49.792621 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:49.792624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.793319 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:49.793321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:49.793741 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:49.793744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.793815 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:49.794284 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:49.794288 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:49.794551 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:49.796049 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:49.796050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:49.796051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:49.796239 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:49.796406 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:49.796407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:49.796408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:49.796453 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:49.796680 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:49.796681 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:49.796782 (+ 101us) client_negotiation.cc:769] Sending connection context
1023 20:11:49.796875 (+ 93us) client_negotiation.cc:240] Negotiation successful
1023 20:11:49.796980 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20251023 20:11:49.797093 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.791931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:42567 (local address 127.9.117.254:40487)
1023 20:11:49.791988 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:49.791990 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:49.792108 (+ 118us) server_negotiation.cc:408] Connection header received
1023 20:11:49.792171 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:49.792172 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:49.792182 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:49.792236 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:49.792765 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.793242 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:49.793818 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.794019 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:49.794629 (+ 610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:49.794632 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:49.794632 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:49.794638 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:49.795952 (+ 1314us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:49.796327 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:49.796328 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:49.796329 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:49.796352 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:49.796499 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:49.796500 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:49.796501 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:49.796614 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:49.796681 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:49.796928 (+ 247us) server_negotiation.cc:300] Negotiation successful
1023 20:11:49.797023 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251023 20:11:49.797298 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 10897
I20251023 20:11:49.797394 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:11:49.797587 9687 external_mini_cluster.cc:1468] Setting key 569f662bb667f263bf3e337c1bf2d1ff
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:49.844061 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40487
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:49.929234 11039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:49.929437 11039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:49.929466 11039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:49.931128 11039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:49.931205 11039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:49.931237 11039 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:11:49.931289 11039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:11:49.933239 11039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:40487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:49.933529 11039 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:49.933748 11039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:49.936581 11047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:49.936581 11045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:49.936556 11044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:49.936767 11039 server_base.cc:1047] running on GCE node
I20251023 20:11:49.937028 11039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:49.937254 11039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:49.938385 11039 hybrid_clock.cc:648] HybridClock initialized: now 1761250309938369 us; error 39 us; skew 500 ppm
Oct 23 20:11:49 dist-test-slave-x62p krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250309, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:49.940800 11039 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:11:49.941354 11039 webserver.cc:492] Webserver started at http://127.9.117.194:44783/ using document root <none> and password file <none>
I20251023 20:11:49.941546 11039 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:49.941591 11039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:49.941707 11039 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:49.942778 11039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "c47b5f1d6b214c07b98d022bd11bed7d"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "cfac445090a8dbf656c39ad38c7f47cf"
server_key_iv: "a81ef3c6d0c7691461691093877c4ce6"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.943091 11039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "c47b5f1d6b214c07b98d022bd11bed7d"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "cfac445090a8dbf656c39ad38c7f47cf"
server_key_iv: "a81ef3c6d0c7691461691093877c4ce6"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.944396 11039 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:11:49.945143 11054 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:49.945333 11039 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:49.945401 11039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "c47b5f1d6b214c07b98d022bd11bed7d"
format_stamp: "Formatted at 2025-10-23 20:11:49 on dist-test-slave-x62p"
server_key: "cfac445090a8dbf656c39ad38c7f47cf"
server_key_iv: "a81ef3c6d0c7691461691093877c4ce6"
server_key_version: "encryptionkey@0"
I20251023 20:11:49.945477 11039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:49.981683 11039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:49.983578 11039 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:49.983695 11039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:49.984290 11039 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:49.984331 11039 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:49.984354 11039 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:49.984367 11039 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 23 20:11:49 dist-test-slave-x62p krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250309, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:49.992209 11039 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:35463
I20251023 20:11:49.992278 11169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:35463 every 8 connection(s)
I20251023 20:11:49.992691 11039 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:11:49.996258 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.984753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:38323 (local address 127.9.117.254:40487)
1023 20:11:49.984819 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:49.984821 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:49.984998 (+ 177us) server_negotiation.cc:408] Connection header received
1023 20:11:49.985574 (+ 576us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:49.985576 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:49.985591 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:49.985651 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:49.986582 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.987594 (+ 1012us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:49.988651 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.988916 (+ 265us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:49.992403 (+ 3487us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:49.992412 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:49.992412 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:49.992424 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:49.994513 (+ 2089us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:49.995117 (+ 604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:49.995119 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:49.995119 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:49.995233 (+ 114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:49.995538 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:49.995540 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:49.995540 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:49.995660 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:49.995735 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:49.996081 (+ 346us) server_negotiation.cc:300] Negotiation successful
1023 20:11:49.996183 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:11:49.996400 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.984713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40487 (local address 127.9.117.194:38323)
1023 20:11:49.984894 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:49.984925 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:49.985459 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:49.985732 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:49.985736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:49.986155 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:49.986463 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:49.986468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.987739 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:49.987741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:49.988541 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:49.988546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:49.988798 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:49.989600 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:49.989653 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:49.992204 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:49.994712 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:49.994718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:49.994720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:49.994973 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:49.995349 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:49.995350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:49.995351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:49.995435 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:49.995759 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:49.995766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:49.995926 (+ 160us) client_negotiation.cc:769] Sending connection context
1023 20:11:49.996040 (+ 114us) client_negotiation.cc:240] Negotiation successful
1023 20:11:49.996184 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20251023 20:11:49.997164 11170 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40487
I20251023 20:11:49.997280 11170 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:49.997588 11170 heartbeater.cc:507] Master 127.9.117.254:40487 requested a full tablet report, sending...
I20251023 20:11:49.998080 10823 ts_manager.cc:194] Registered new tserver with Master: c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463)
I20251023 20:11:49.998653 10823 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:38323
I20251023 20:11:49.999225 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 11039
I20251023 20:11:49.999316 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:11:49.999523 9687 external_mini_cluster.cc:1468] Setting key e5866e7aba82f1dc7ce9b0f9a6556de5
I20251023 20:11:50.004508 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.998355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40487 (local address 127.9.117.194:35427)
1023 20:11:49.998486 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:49.998498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:49.998571 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:49.998690 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:49.998691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:49.999026 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:49.999323 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:49.999327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.000160 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.000162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.000615 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.000618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.000681 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.001251 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:50.001255 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:50.001535 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:50.003564 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:50.003565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:50.003566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:50.003756 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:50.003940 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:50.003940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:50.003941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:50.003965 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:50.004239 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:50.004239 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:50.004278 (+ 39us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.004327 (+ 49us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.004417 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20251023 20:11:50.004608 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:49.998427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:35427 (local address 127.9.117.254:40487)
1023 20:11:49.998479 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:49.998482 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:49.998555 (+ 73us) server_negotiation.cc:408] Connection header received
1023 20:11:49.998628 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:49.998629 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:49.998638 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:49.998684 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:49.999434 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.000058 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.000702 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.000931 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.001647 (+ 716us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:50.001651 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:50.001651 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:50.001660 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:50.003474 (+ 1814us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.003841 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.003842 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.003843 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.003872 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.004011 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.004011 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.004012 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.004180 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:50.004257 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.004372 (+ 115us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.004470 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:50.045884 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40487
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:50.131301 11181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:50.131500 11181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:50.131541 11181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:50.133121 11181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:50.133172 11181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:50.133208 11181 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:11:50.133241 11181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:11:50.134944 11181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:40487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:50.135249 11181 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:50.135447 11181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:50.138814 11189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:50.138808 11186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:50.138813 11187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:50.138850 11181 server_base.cc:1047] running on GCE node
I20251023 20:11:50.139334 11181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:50.139632 11181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:50.140800 11181 hybrid_clock.cc:648] HybridClock initialized: now 1761250310140773 us; error 45 us; skew 500 ppm
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250310, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:50.143415 11181 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:11:50.144057 11181 webserver.cc:492] Webserver started at http://127.9.117.195:37427/ using document root <none> and password file <none>
I20251023 20:11:50.144308 11181 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:50.144353 11181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:50.144492 11181 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:50.146013 11181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "864df332d01b445baaa4dd7cfa37d755"
format_stamp: "Formatted at 2025-10-23 20:11:50 on dist-test-slave-x62p"
server_key: "ebcd3e01e85f8031f25bdc41eee2950c"
server_key_iv: "dd84dcf7fe125ee7ac25241bc16a83e9"
server_key_version: "encryptionkey@0"
I20251023 20:11:50.146414 11181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "864df332d01b445baaa4dd7cfa37d755"
format_stamp: "Formatted at 2025-10-23 20:11:50 on dist-test-slave-x62p"
server_key: "ebcd3e01e85f8031f25bdc41eee2950c"
server_key_iv: "dd84dcf7fe125ee7ac25241bc16a83e9"
server_key_version: "encryptionkey@0"
I20251023 20:11:50.148123 11181 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20251023 20:11:50.149299 11196 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:50.150434 11181 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251023 20:11:50.150609 11181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "864df332d01b445baaa4dd7cfa37d755"
format_stamp: "Formatted at 2025-10-23 20:11:50 on dist-test-slave-x62p"
server_key: "ebcd3e01e85f8031f25bdc41eee2950c"
server_key_iv: "dd84dcf7fe125ee7ac25241bc16a83e9"
server_key_version: "encryptionkey@0"
I20251023 20:11:50.150724 11181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:50.160621 11181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:50.162545 11181 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:50.162695 11181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:50.163285 11181 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:50.163327 11181 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:50.163367 11181 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:50.163398 11181 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250310, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:50.171473 11181 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:35987
I20251023 20:11:50.171634 11311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:35987 every 8 connection(s)
I20251023 20:11:50.172003 11181 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:11:50.174121 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.163729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:60821 (local address 127.9.117.254:40487)
1023 20:11:50.163804 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.163808 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.164020 (+ 212us) server_negotiation.cc:408] Connection header received
1023 20:11:50.164561 (+ 541us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.164563 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.164580 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.164666 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:50.165778 (+ 1112us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.166576 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.167622 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.167962 (+ 340us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.170737 (+ 2775us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:50.170743 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:50.170743 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:50.170755 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:50.172636 (+ 1881us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.173201 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.173202 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.173203 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.173234 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.173408 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.173408 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.173408 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.173517 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:50.173567 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.173958 (+ 391us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.174051 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251023 20:11:50.174180 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.163693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40487 (local address 127.9.117.195:60821)
1023 20:11:50.163904 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.163935 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.164464 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.164721 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.164745 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.165167 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:50.165648 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.165654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.166687 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.166690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.167452 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.167457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.167686 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.168409 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:50.168448 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:50.170611 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:50.172796 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:50.172805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:50.172808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:50.173094 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:50.173297 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:50.173297 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:50.173298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:50.173358 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:50.173601 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:50.173607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:50.173772 (+ 165us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.173878 (+ 106us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.174011 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":127,"threads_started":1}
I20251023 20:11:50.174769 11312 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40487
I20251023 20:11:50.174862 11312 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:50.175081 11312 heartbeater.cc:507] Master 127.9.117.254:40487 requested a full tablet report, sending...
I20251023 20:11:50.175519 10822 ts_manager.cc:194] Registered new tserver with Master: 864df332d01b445baaa4dd7cfa37d755 (127.9.117.195:35987)
I20251023 20:11:50.175957 10822 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:60821
I20251023 20:11:50.181321 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 11181
I20251023 20:11:50.181432 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:11:50.181613 9687 external_mini_cluster.cc:1468] Setting key c1e7142bc275aa1bd871f66bc4c8bf26
I20251023 20:11:50.182183 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.176306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:39667 (local address 127.9.117.254:40487)
1023 20:11:50.176386 (+ 80us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.176388 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.176532 (+ 144us) server_negotiation.cc:408] Connection header received
1023 20:11:50.176564 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.176564 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.176577 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.176642 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:50.177129 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.177622 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.178238 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.178360 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.179192 (+ 832us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:50.179196 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:50.179196 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:50.179203 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:50.180650 (+ 1447us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.181146 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.181147 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.181148 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.181177 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.181456 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.181458 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.181458 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.181597 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:50.181788 (+ 191us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.181966 (+ 178us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.182078 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251023 20:11:50.182199 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.176219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40487 (local address 127.9.117.195:39667)
1023 20:11:50.176449 (+ 230us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.176459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.176522 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.176644 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.176645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.176865 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:50.177050 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.177053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.177742 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.177743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.178156 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.178158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.178225 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.178787 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:50.178792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:50.179100 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:50.180810 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:50.180811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:50.180812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:50.180972 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:50.181275 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:50.181277 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:50.181278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:50.181321 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:50.181768 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:50.181770 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:50.181834 (+ 64us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.181922 (+ 88us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.182079 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
I20251023 20:11:50.182787 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:50.205627 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.198765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.9.117.254:40487)
1023 20:11:50.198832 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.198835 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.198844 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:11:50.198933 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.198934 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.198947 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.199000 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:50.199723 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.200229 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.200852 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.201058 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.203234 (+ 2176us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:50.203240 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:50.203240 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:50.203249 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:50.204590 (+ 1341us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.205004 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.205004 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.205005 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.205027 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.205206 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.205207 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.205207 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.205302 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:50.205379 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.205464 (+ 85us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.205538 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:11:50.207440 10823 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:11:50.208864 10823 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:50.214514 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.211431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:35463 (local address 127.0.0.1:41834)
1023 20:11:50.211705 (+ 274us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.211714 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.211767 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.211991 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.211992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.211995 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:50.212246 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.212249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.213278 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.213280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.214155 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.214157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.214235 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.214284 (+ 49us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.214340 (+ 56us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.214387 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":58,"threads_started":1}
I20251023 20:11:50.214514 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.211274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:35987 (local address 127.0.0.1:51150)
1023 20:11:50.211544 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.211554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.211611 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.211820 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.211823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.211826 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:50.212120 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.212124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.213442 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.213443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.214246 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.214248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.214314 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.214317 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.214352 (+ 35us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.214387 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":78,"threads_started":1}
I20251023 20:11:50.215005 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.211040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:40723 (local address 127.0.0.1:49064)
1023 20:11:50.211182 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.211212 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.211297 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.212572 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.212573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.212576 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:50.212793 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.212794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.213946 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.213948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.214808 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.214820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.214874 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.214878 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.214903 (+ 25us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.214938 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20251023 20:11:50.215018 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.211553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.9.117.194:35463)
1023 20:11:50.211698 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.211704 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.211757 (+ 53us) server_negotiation.cc:408] Connection header received
1023 20:11:50.211836 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.211839 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.211857 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.211913 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:50.212314 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.213188 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.214268 (+ 1080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.214791 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.214857 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.214890 (+ 33us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.214938 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20251023 20:11:50.215257 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.211380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.9.117.195:35987)
1023 20:11:50.211588 (+ 208us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.211597 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.211610 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:11:50.211691 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.211695 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.211715 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.211790 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:50.212198 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.213365 (+ 1167us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.214570 (+ 1205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.215051 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.215101 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.215138 (+ 37us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.215179 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20251023 20:11:50.215723 11104 tablet_service.cc:1505] Processing CreateTablet for tablet fad055c927f9473f898c123fbc44209d (DEFAULT_TABLE table=test-table [id=079f9a86250f49ad8ca61a8ae99ec40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:50.215814 11246 tablet_service.cc:1505] Processing CreateTablet for tablet fad055c927f9473f898c123fbc44209d (DEFAULT_TABLE table=test-table [id=079f9a86250f49ad8ca61a8ae99ec40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:50.216046 11104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad055c927f9473f898c123fbc44209d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.216125 11246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad055c927f9473f898c123fbc44209d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.216212 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.211150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.9.117.193:40723)
1023 20:11:50.212408 (+ 1258us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.212415 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.212425 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:11:50.212466 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.212468 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.212483 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.212547 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:50.212867 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.213798 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.215456 (+ 1658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.215996 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.216043 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.216075 (+ 32us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.216123 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":97,"threads_started":1}
I20251023 20:11:50.217366 10962 tablet_service.cc:1505] Processing CreateTablet for tablet fad055c927f9473f898c123fbc44209d (DEFAULT_TABLE table=test-table [id=079f9a86250f49ad8ca61a8ae99ec40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:50.217676 10962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad055c927f9473f898c123fbc44209d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.218662 11335 tablet_bootstrap.cc:492] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Bootstrap starting.
I20251023 20:11:50.219230 11336 tablet_bootstrap.cc:492] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Bootstrap starting.
I20251023 20:11:50.219419 11335 tablet_bootstrap.cc:654] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.219975 11336 tablet_bootstrap.cc:654] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.220081 11335 log.cc:826] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:50.220501 11337 tablet_bootstrap.cc:492] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Bootstrap starting.
I20251023 20:11:50.220623 11336 log.cc:826] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:50.220948 11335 tablet_bootstrap.cc:492] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: No bootstrap required, opened a new log
I20251023 20:11:50.221012 11335 ts_tablet_manager.cc:1403] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:11:50.221508 11336 tablet_bootstrap.cc:492] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: No bootstrap required, opened a new log
I20251023 20:11:50.221675 11336 ts_tablet_manager.cc:1403] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:11:50.222673 11335 raft_consensus.cc:359] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.222862 11335 raft_consensus.cc:385] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.222908 11335 raft_consensus.cc:740] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864df332d01b445baaa4dd7cfa37d755, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.223019 11335 consensus_queue.cc:260] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.223286 11335 ts_tablet_manager.cc:1434] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:11:50.223328 11312 heartbeater.cc:499] Master 127.9.117.254:40487 was elected leader, sending a full tablet report...
I20251023 20:11:50.223496 11336 raft_consensus.cc:359] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.223666 11336 raft_consensus.cc:385] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.224069 11337 tablet_bootstrap.cc:654] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.224117 11336 raft_consensus.cc:740] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47b5f1d6b214c07b98d022bd11bed7d, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.224239 11336 consensus_queue.cc:260] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.224709 11336 ts_tablet_manager.cc:1434] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:11:50.224710 11337 log.cc:826] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:50.224900 11170 heartbeater.cc:499] Master 127.9.117.254:40487 was elected leader, sending a full tablet report...
I20251023 20:11:50.225685 11337 tablet_bootstrap.cc:492] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: No bootstrap required, opened a new log
I20251023 20:11:50.225769 11337 ts_tablet_manager.cc:1403] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20251023 20:11:50.226949 11337 raft_consensus.cc:359] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.227077 11337 raft_consensus.cc:385] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.227097 11337 raft_consensus.cc:740] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17482232f614f468266b744a03b19e0, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.227200 11337 consensus_queue.cc:260] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.227427 11337 ts_tablet_manager.cc:1434] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:11:50.227485 11028 heartbeater.cc:499] Master 127.9.117.254:40487 was elected leader, sending a full tablet report...
W20251023 20:11:50.243396 11171 tablet.cc:2378] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:11:50.288182 11029 tablet.cc:2378] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:11:50.422748 11313 tablet.cc:2378] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:11:50.483768 11340 raft_consensus.cc:493] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:50.483922 11340 raft_consensus.cc:515] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.484252 11340 leader_election.cc:290] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a17482232f614f468266b744a03b19e0 (127.9.117.193:40723), c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463)
I20251023 20:11:50.486840 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.484469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:40723 (local address 127.9.117.195:59457)
1023 20:11:50.484535 (+ 66us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.484548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.484599 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.484862 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.484863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.484864 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:50.485082 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.485084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.485971 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.485972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.486653 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.486655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.486713 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.486718 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.486742 (+ 24us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.486770 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20251023 20:11:50.487360 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.484562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:59457 (local address 127.9.117.193:40723)
1023 20:11:50.484660 (+ 98us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.484663 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.484676 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:11:50.484712 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.484713 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.484748 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.484844 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:50.485198 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.485882 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.486757 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.487225 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.487235 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.487256 (+ 21us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.487289 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251023 20:11:50.487713 11344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.484628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:35463 (local address 127.9.117.195:42315)
1023 20:11:50.484824 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.484835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.484906 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.485163 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.485165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.485168 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:50.485437 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.485439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.486261 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.486262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.486915 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.486917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.487539 (+ 622us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.487542 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.487592 (+ 50us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.487637 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":131,"threads_started":1}
I20251023 20:11:50.487730 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.484691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:42315 (local address 127.9.117.194:35463)
1023 20:11:50.484784 (+ 93us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.484786 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.484911 (+ 125us) server_negotiation.cc:408] Connection header received
1023 20:11:50.485070 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.485072 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.485082 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.485144 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:50.485547 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.486178 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.487026 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.487503 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.487511 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.487635 (+ 124us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.487681 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251023 20:11:50.487746 10982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fad055c927f9473f898c123fbc44209d" candidate_uuid: "864df332d01b445baaa4dd7cfa37d755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17482232f614f468266b744a03b19e0" is_pre_election: true
I20251023 20:11:50.487917 10982 raft_consensus.cc:2468] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864df332d01b445baaa4dd7cfa37d755 in term 0.
I20251023 20:11:50.488065 11124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fad055c927f9473f898c123fbc44209d" candidate_uuid: "864df332d01b445baaa4dd7cfa37d755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" is_pre_election: true
I20251023 20:11:50.488166 11198 leader_election.cc:304] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 864df332d01b445baaa4dd7cfa37d755, a17482232f614f468266b744a03b19e0; no voters:
I20251023 20:11:50.488216 11124 raft_consensus.cc:2468] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864df332d01b445baaa4dd7cfa37d755 in term 0.
I20251023 20:11:50.488307 11340 raft_consensus.cc:2804] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:50.488358 11340 raft_consensus.cc:493] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:50.488386 11340 raft_consensus.cc:3060] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.489248 11340 raft_consensus.cc:515] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.489398 11340 leader_election.cc:290] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [CANDIDATE]: Term 1 election: Requested vote from peers a17482232f614f468266b744a03b19e0 (127.9.117.193:40723), c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463)
I20251023 20:11:50.489567 10982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fad055c927f9473f898c123fbc44209d" candidate_uuid: "864df332d01b445baaa4dd7cfa37d755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17482232f614f468266b744a03b19e0"
I20251023 20:11:50.489598 11124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fad055c927f9473f898c123fbc44209d" candidate_uuid: "864df332d01b445baaa4dd7cfa37d755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47b5f1d6b214c07b98d022bd11bed7d"
I20251023 20:11:50.489653 10982 raft_consensus.cc:3060] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.489663 11124 raft_consensus.cc:3060] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.490334 10982 raft_consensus.cc:2468] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864df332d01b445baaa4dd7cfa37d755 in term 1.
I20251023 20:11:50.490345 11124 raft_consensus.cc:2468] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864df332d01b445baaa4dd7cfa37d755 in term 1.
I20251023 20:11:50.490516 11198 leader_election.cc:304] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 864df332d01b445baaa4dd7cfa37d755, c47b5f1d6b214c07b98d022bd11bed7d; no voters:
I20251023 20:11:50.490610 11340 raft_consensus.cc:2804] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:50.490793 11340 raft_consensus.cc:697] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 1 LEADER]: Becoming Leader. State: Replica: 864df332d01b445baaa4dd7cfa37d755, State: Running, Role: LEADER
I20251023 20:11:50.490881 11340 consensus_queue.cc:237] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } }
I20251023 20:11:50.491678 10823 catalog_manager.cc:5649] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 864df332d01b445baaa4dd7cfa37d755 (127.9.117.195). New cstate: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } health_report { overall_health: UNKNOWN } } }
I20251023 20:11:50.502234 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.500074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.9.117.195:35987)
1023 20:11:50.500139 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.500141 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.500150 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:11:50.500191 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.500192 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.500208 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.500262 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:50.500528 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.501074 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.501660 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.501843 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.501879 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:50.501978 (+ 99us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:50.502048 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.502117 (+ 69us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.502145 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251023 20:11:50.504057 10982 raft_consensus.cc:1275] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Refusing update from remote peer 864df332d01b445baaa4dd7cfa37d755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:50.504057 11124 raft_consensus.cc:1275] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Refusing update from remote peer 864df332d01b445baaa4dd7cfa37d755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:50.504396 11340 consensus_queue.cc:1048] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:50.504532 11345 consensus_queue.cc:1048] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:50.507234 11348 mvcc.cc:204] Tried to move back new op lower bound from 7214081271821737984 to 7214081271771103232. Current Snapshot: MvccSnapshot[applied={T|T < 7214081271821737984}]
I20251023 20:11:50.507603 11349 mvcc.cc:204] Tried to move back new op lower bound from 7214081271821737984 to 7214081271771103232. Current Snapshot: MvccSnapshot[applied={T|T < 7214081271821737984}]
I20251023 20:11:50.508179 11350 mvcc.cc:204] Tried to move back new op lower bound from 7214081271821737984 to 7214081271771103232. Current Snapshot: MvccSnapshot[applied={T|T < 7214081271821737984}]
I20251023 20:11:50.511221 10823 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45280:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:50.511336 10823 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45280:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:50.512169 10823 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec: Sending DeleteTablet for 3 replicas of tablet fad055c927f9473f898c123fbc44209d
I20251023 20:11:50.512540 11246 tablet_service.cc:1552] Processing DeleteTablet for tablet fad055c927f9473f898c123fbc44209d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:50 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:51150
I20251023 20:11:50.512603 10962 tablet_service.cc:1552] Processing DeleteTablet for tablet fad055c927f9473f898c123fbc44209d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:50 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:49064
I20251023 20:11:50.512845 11359 tablet_replica.cc:333] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: stopping tablet replica
I20251023 20:11:50.512825 11104 tablet_service.cc:1552] Processing DeleteTablet for tablet fad055c927f9473f898c123fbc44209d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:50 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:41834
I20251023 20:11:50.512928 11359 raft_consensus.cc:2243] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:50.513033 11359 raft_consensus.cc:2272] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:50.513075 10823 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:11:50.513238 10823 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:50.513484 11359 ts_tablet_manager.cc:1916] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:50.513679 11358 tablet_replica.cc:333] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: stopping tablet replica
I20251023 20:11:50.513862 11360 tablet_replica.cc:333] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: stopping tablet replica
I20251023 20:11:50.513994 11358 raft_consensus.cc:2243] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:11:50.514119 11360 raft_consensus.cc:2243] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:50.514269 11360 raft_consensus.cc:2272] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:50.514443 11358 raft_consensus.cc:2272] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:50.515054 11246 tablet_service.cc:1505] Processing CreateTablet for tablet e057cd796db9434fb64d9ef7bc6467a6 (DEFAULT_TABLE table=test-table [id=af60dddb34a04d28ba15ecfef7e5c67d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:50.515091 11360 ts_tablet_manager.cc:1916] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:50.515201 11246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e057cd796db9434fb64d9ef7bc6467a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.515350 11359 ts_tablet_manager.cc:1929] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:11:50.515413 11359 log.cc:1199] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/fad055c927f9473f898c123fbc44209d
I20251023 20:11:50.515650 11359 ts_tablet_manager.cc:1950] T fad055c927f9473f898c123fbc44209d P a17482232f614f468266b744a03b19e0: Deleting consensus metadata
I20251023 20:11:50.516228 10808 catalog_manager.cc:4985] TS a17482232f614f468266b744a03b19e0 (127.9.117.193:40723): tablet fad055c927f9473f898c123fbc44209d (table test-table [id=079f9a86250f49ad8ca61a8ae99ec40b]) successfully deleted
I20251023 20:11:50.516862 11335 tablet_bootstrap.cc:492] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: Bootstrap starting.
I20251023 20:11:50.516870 11360 ts_tablet_manager.cc:1929] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:11:50.517194 10962 tablet_service.cc:1505] Processing CreateTablet for tablet e057cd796db9434fb64d9ef7bc6467a6 (DEFAULT_TABLE table=test-table [id=af60dddb34a04d28ba15ecfef7e5c67d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:50.517306 10962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e057cd796db9434fb64d9ef7bc6467a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.516934 11360 log.cc:1199] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/fad055c927f9473f898c123fbc44209d
I20251023 20:11:50.517436 11335 tablet_bootstrap.cc:654] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.517722 11360 ts_tablet_manager.cc:1950] T fad055c927f9473f898c123fbc44209d P c47b5f1d6b214c07b98d022bd11bed7d: Deleting consensus metadata
I20251023 20:11:50.517972 11104 tablet_service.cc:1505] Processing CreateTablet for tablet e057cd796db9434fb64d9ef7bc6467a6 (DEFAULT_TABLE table=test-table [id=af60dddb34a04d28ba15ecfef7e5c67d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:50.518075 11104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e057cd796db9434fb64d9ef7bc6467a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.518673 11337 tablet_bootstrap.cc:492] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: Bootstrap starting.
I20251023 20:11:50.519182 11337 tablet_bootstrap.cc:654] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.519364 10808 catalog_manager.cc:4985] TS c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463): tablet fad055c927f9473f898c123fbc44209d (table test-table [id=079f9a86250f49ad8ca61a8ae99ec40b]) successfully deleted
I20251023 20:11:50.517416 11358 ts_tablet_manager.cc:1916] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:50.519519 11336 tablet_bootstrap.cc:492] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: Bootstrap starting.
I20251023 20:11:50.519937 11336 tablet_bootstrap.cc:654] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.520491 11337 tablet_bootstrap.cc:492] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: No bootstrap required, opened a new log
I20251023 20:11:50.520547 11337 ts_tablet_manager.cc:1403] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:11:50.521018 11336 tablet_bootstrap.cc:492] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: No bootstrap required, opened a new log
I20251023 20:11:50.521066 11336 ts_tablet_manager.cc:1403] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:11:50.521158 11335 tablet_bootstrap.cc:492] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: No bootstrap required, opened a new log
I20251023 20:11:50.521210 11335 ts_tablet_manager.cc:1403] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:11:50.521227 11336 raft_consensus.cc:359] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.521296 11336 raft_consensus.cc:385] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.521317 11336 raft_consensus.cc:740] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47b5f1d6b214c07b98d022bd11bed7d, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.521361 11336 consensus_queue.cc:260] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.521440 11336 ts_tablet_manager.cc:1434] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:50.521401 11335 raft_consensus.cc:359] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.521472 11335 raft_consensus.cc:385] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.521520 11335 raft_consensus.cc:740] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864df332d01b445baaa4dd7cfa37d755, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.521569 11335 consensus_queue.cc:260] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.521674 11335 ts_tablet_manager.cc:1434] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251023 20:11:50.521458 11358 ts_tablet_manager.cc:1929] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:11:50.522053 11358 log.cc:1199] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/fad055c927f9473f898c123fbc44209d
I20251023 20:11:50.522338 11358 ts_tablet_manager.cc:1950] T fad055c927f9473f898c123fbc44209d P 864df332d01b445baaa4dd7cfa37d755: Deleting consensus metadata
I20251023 20:11:50.520721 11337 raft_consensus.cc:359] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.522380 11337 raft_consensus.cc:385] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.522419 11337 raft_consensus.cc:740] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17482232f614f468266b744a03b19e0, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.522477 11337 consensus_queue.cc:260] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.522567 11337 ts_tablet_manager.cc:1434] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:11:50.522732 10808 catalog_manager.cc:4985] TS 864df332d01b445baaa4dd7cfa37d755 (127.9.117.195:35987): tablet fad055c927f9473f898c123fbc44209d (table test-table [id=079f9a86250f49ad8ca61a8ae99ec40b]) successfully deleted
I20251023 20:11:50.720273 11343 raft_consensus.cc:493] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:50.720393 11343 raft_consensus.cc:515] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.720774 11343 leader_election.cc:290] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463), 864df332d01b445baaa4dd7cfa37d755 (127.9.117.195:35987)
I20251023 20:11:50.723349 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.721020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:35463 (local address 127.9.117.193:44307)
1023 20:11:50.721153 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.721166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.721246 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.721501 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.721502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.721505 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:50.721805 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.721807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.722518 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.722519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.723164 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.723166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.723227 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.723231 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.723249 (+ 18us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.723282 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20251023 20:11:50.723853 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.721127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:44307 (local address 127.9.117.194:35463)
1023 20:11:50.721184 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.721186 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.721226 (+ 40us) server_negotiation.cc:408] Connection header received
1023 20:11:50.721357 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.721358 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.721371 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.721404 (+ 33us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:50.721905 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.722416 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.723274 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.723707 (+ 433us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.723717 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.723739 (+ 22us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.723776 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:11:50.723870 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.721218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:35987 (local address 127.9.117.193:50893)
1023 20:11:50.721345 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.721355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.721434 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.721637 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.721638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.721641 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:50.721875 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.721877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.722592 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.722593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.723649 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.723651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.723724 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.723730 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.723764 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.723795 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20251023 20:11:50.724079 11124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e057cd796db9434fb64d9ef7bc6467a6" candidate_uuid: "a17482232f614f468266b744a03b19e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" is_pre_election: true
I20251023 20:11:50.724162 11124 raft_consensus.cc:2468] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a17482232f614f468266b744a03b19e0 in term 0.
I20251023 20:11:50.724344 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.721306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:50893 (local address 127.9.117.195:35987)
1023 20:11:50.721374 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.721376 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.721438 (+ 62us) server_negotiation.cc:408] Connection header received
1023 20:11:50.721517 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.721518 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.721528 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.721590 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:50.721994 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.722477 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.723744 (+ 1267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.724201 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.724210 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.724233 (+ 23us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.724272 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:11:50.724367 10914 leader_election.cc:304] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17482232f614f468266b744a03b19e0, c47b5f1d6b214c07b98d022bd11bed7d; no voters:
I20251023 20:11:50.724538 11343 raft_consensus.cc:2804] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:50.724589 11343 raft_consensus.cc:493] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:50.724604 11343 raft_consensus.cc:3060] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.724571 11266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e057cd796db9434fb64d9ef7bc6467a6" candidate_uuid: "a17482232f614f468266b744a03b19e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "864df332d01b445baaa4dd7cfa37d755" is_pre_election: true
I20251023 20:11:50.724694 11266 raft_consensus.cc:2468] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a17482232f614f468266b744a03b19e0 in term 0.
I20251023 20:11:50.725368 11343 raft_consensus.cc:515] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.725500 11343 leader_election.cc:290] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [CANDIDATE]: Term 1 election: Requested vote from peers c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463), 864df332d01b445baaa4dd7cfa37d755 (127.9.117.195:35987)
I20251023 20:11:50.725683 11124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e057cd796db9434fb64d9ef7bc6467a6" candidate_uuid: "a17482232f614f468266b744a03b19e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47b5f1d6b214c07b98d022bd11bed7d"
I20251023 20:11:50.725751 11124 raft_consensus.cc:3060] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.725736 11266 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e057cd796db9434fb64d9ef7bc6467a6" candidate_uuid: "a17482232f614f468266b744a03b19e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "864df332d01b445baaa4dd7cfa37d755"
I20251023 20:11:50.725816 11266 raft_consensus.cc:3060] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.726445 11124 raft_consensus.cc:2468] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17482232f614f468266b744a03b19e0 in term 1.
I20251023 20:11:50.726550 11266 raft_consensus.cc:2468] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17482232f614f468266b744a03b19e0 in term 1.
I20251023 20:11:50.726648 10914 leader_election.cc:304] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17482232f614f468266b744a03b19e0, c47b5f1d6b214c07b98d022bd11bed7d; no voters:
I20251023 20:11:50.726744 11343 raft_consensus.cc:2804] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:50.726888 11343 raft_consensus.cc:697] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 1 LEADER]: Becoming Leader. State: Replica: a17482232f614f468266b744a03b19e0, State: Running, Role: LEADER
I20251023 20:11:50.726948 11343 consensus_queue.cc:237] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.727669 10822 catalog_manager.cc:5649] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a17482232f614f468266b744a03b19e0 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "a17482232f614f468266b744a03b19e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } health_report { overall_health: UNKNOWN } } }
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250309, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:50.768323 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.761609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.9.117.254:40487)
1023 20:11:50.761747 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.761750 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.761825 (+ 75us) server_negotiation.cc:408] Connection header received
1023 20:11:50.761895 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.761896 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.761908 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.761976 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:50.762658 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.763201 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.763725 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.763896 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.766023 (+ 2127us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:50.766027 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:50.766027 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:50.766035 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:50.767285 (+ 1250us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.767652 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.767653 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.767653 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.767676 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.767807 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.767808 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.767808 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.767908 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:50.767970 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.768101 (+ 131us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.768195 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20251023 20:11:50.768378 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.761531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40487 (local address 127.0.0.1:45292)
1023 20:11:50.761710 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.761724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.761826 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.761985 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.761986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.762377 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:50.762554 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.762557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.763282 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.763283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.763651 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.763652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.763721 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.764377 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:50.764382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:50.765934 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:50.767370 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:50.767378 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:50.767382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:50.767586 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:50.767729 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:50.767730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:50.767730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:50.767758 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:50.767961 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:50.767968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:50.768014 (+ 46us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.768074 (+ 60us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.768265 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":81,"thread_start_us":111,"threads_started":1}
I20251023 20:11:50.770326 10822 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:45292:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251023 20:11:50.770532 10822 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:50.772210 10962 tablet_service.cc:1505] Processing CreateTablet for tablet eda93ccb97bf4f15be1cfe9a72b9ac88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f736516fb1d844caa52cb6b54fb442f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:11:50.772229 11104 tablet_service.cc:1505] Processing CreateTablet for tablet eda93ccb97bf4f15be1cfe9a72b9ac88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f736516fb1d844caa52cb6b54fb442f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:11:50.772244 11246 tablet_service.cc:1505] Processing CreateTablet for tablet eda93ccb97bf4f15be1cfe9a72b9ac88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f736516fb1d844caa52cb6b54fb442f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:11:50.772389 11104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda93ccb97bf4f15be1cfe9a72b9ac88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.772389 10962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda93ccb97bf4f15be1cfe9a72b9ac88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.772389 11246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda93ccb97bf4f15be1cfe9a72b9ac88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:50.773664 11336 tablet_bootstrap.cc:492] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: Bootstrap starting.
I20251023 20:11:50.773991 11337 tablet_bootstrap.cc:492] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: Bootstrap starting.
I20251023 20:11:50.774001 11335 tablet_bootstrap.cc:492] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: Bootstrap starting.
I20251023 20:11:50.774324 11335 tablet_bootstrap.cc:654] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.774335 11336 tablet_bootstrap.cc:654] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.774506 11337 tablet_bootstrap.cc:654] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:50.774973 11335 tablet_bootstrap.cc:492] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: No bootstrap required, opened a new log
I20251023 20:11:50.775038 11335 ts_tablet_manager.cc:1403] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:50.775182 11335 raft_consensus.cc:359] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.775261 11335 raft_consensus.cc:385] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.775281 11335 raft_consensus.cc:740] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864df332d01b445baaa4dd7cfa37d755, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.775270 11337 tablet_bootstrap.cc:492] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: No bootstrap required, opened a new log
I20251023 20:11:50.775323 11337 ts_tablet_manager.cc:1403] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:50.775331 11335 consensus_queue.cc:260] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.775449 11335 ts_tablet_manager.cc:1434] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251023 20:11:50.775465 11353 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.775533 11353 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:50.775540 11337 raft_consensus.cc:359] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.775631 11337 raft_consensus.cc:385] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.775656 11337 raft_consensus.cc:740] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17482232f614f468266b744a03b19e0, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.775710 11337 consensus_queue.cc:260] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.775836 11337 ts_tablet_manager.cc:1434] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:50.775838 11343 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.775964 11343 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:50.776273 11336 tablet_bootstrap.cc:492] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: No bootstrap required, opened a new log
I20251023 20:11:50.776319 11336 ts_tablet_manager.cc:1403] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251023 20:11:50.776479 11336 raft_consensus.cc:359] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.776535 11336 raft_consensus.cc:385] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:50.776557 11336 raft_consensus.cc:740] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47b5f1d6b214c07b98d022bd11bed7d, State: Initialized, Role: FOLLOWER
I20251023 20:11:50.776623 11336 consensus_queue.cc:260] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.776719 11336 ts_tablet_manager.cc:1434] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:50.776801 11341 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.776901 11341 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:50.807186 11353 raft_consensus.cc:493] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:50.807324 11353 raft_consensus.cc:515] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.807534 11353 leader_election.cc:290] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a17482232f614f468266b744a03b19e0 (127.9.117.193:40723), c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463)
I20251023 20:11:50.807996 10982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eda93ccb97bf4f15be1cfe9a72b9ac88" candidate_uuid: "864df332d01b445baaa4dd7cfa37d755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17482232f614f468266b744a03b19e0" is_pre_election: true
I20251023 20:11:50.807998 11124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eda93ccb97bf4f15be1cfe9a72b9ac88" candidate_uuid: "864df332d01b445baaa4dd7cfa37d755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" is_pre_election: true
I20251023 20:11:50.808104 11124 raft_consensus.cc:2468] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864df332d01b445baaa4dd7cfa37d755 in term 0.
I20251023 20:11:50.808109 10982 raft_consensus.cc:2468] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864df332d01b445baaa4dd7cfa37d755 in term 0.
I20251023 20:11:50.808313 11198 leader_election.cc:304] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 864df332d01b445baaa4dd7cfa37d755, c47b5f1d6b214c07b98d022bd11bed7d; no voters:
I20251023 20:11:50.808455 11353 raft_consensus.cc:2804] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:50.808519 11353 raft_consensus.cc:493] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:50.808538 11353 raft_consensus.cc:3060] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.809643 11353 raft_consensus.cc:515] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.809789 11353 leader_election.cc:290] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [CANDIDATE]: Term 1 election: Requested vote from peers a17482232f614f468266b744a03b19e0 (127.9.117.193:40723), c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463)
I20251023 20:11:50.810019 10982 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eda93ccb97bf4f15be1cfe9a72b9ac88" candidate_uuid: "864df332d01b445baaa4dd7cfa37d755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17482232f614f468266b744a03b19e0"
I20251023 20:11:50.810032 11124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eda93ccb97bf4f15be1cfe9a72b9ac88" candidate_uuid: "864df332d01b445baaa4dd7cfa37d755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47b5f1d6b214c07b98d022bd11bed7d"
I20251023 20:11:50.810115 11124 raft_consensus.cc:3060] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.810115 10982 raft_consensus.cc:3060] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:50.810976 10982 raft_consensus.cc:2468] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864df332d01b445baaa4dd7cfa37d755 in term 1.
I20251023 20:11:50.810968 11124 raft_consensus.cc:2468] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864df332d01b445baaa4dd7cfa37d755 in term 1.
I20251023 20:11:50.811282 11198 leader_election.cc:304] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 864df332d01b445baaa4dd7cfa37d755, c47b5f1d6b214c07b98d022bd11bed7d; no voters:
I20251023 20:11:50.811412 11353 raft_consensus.cc:2804] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:50.811586 11353 raft_consensus.cc:697] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [term 1 LEADER]: Becoming Leader. State: Replica: 864df332d01b445baaa4dd7cfa37d755, State: Running, Role: LEADER
I20251023 20:11:50.811686 11353 consensus_queue.cc:237] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } }
I20251023 20:11:50.812003 11345 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: TxnStatusTablet state changed. Reason: New leader 864df332d01b445baaa4dd7cfa37d755. Latest consensus state: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.812083 11345 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:11:50.812216 11379 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251023 20:11:50.812322 11379 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251023 20:11:50.812492 10822 catalog_manager.cc:5649] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 864df332d01b445baaa4dd7cfa37d755 (127.9.117.195). New cstate: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } health_report { overall_health: HEALTHY } } }
I20251023 20:11:50.823899 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.821442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:35987 (local address 127.0.0.1:51174)
1023 20:11:50.821575 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.821589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.821647 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.821833 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.821834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.821836 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:11:50.821996 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.821998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.822907 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.822908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.823337 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.823338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.823401 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.823423 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:11:50.823725 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:11:50.823735 (+ 10us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.823781 (+ 46us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.823834 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20251023 20:11:50.823910 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.821538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.9.117.195:35987)
1023 20:11:50.821611 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.821614 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.821626 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:11:50.821713 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.821714 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.821728 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.821795 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:50.822068 (+ 273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.822814 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.823412 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.823598 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.823620 (+ 22us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:50.823656 (+ 36us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:50.823716 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.823798 (+ 82us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.823822 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251023 20:11:50.825021 11124 raft_consensus.cc:1275] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Refusing update from remote peer 864df332d01b445baaa4dd7cfa37d755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:50.825058 10982 raft_consensus.cc:1275] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Refusing update from remote peer 864df332d01b445baaa4dd7cfa37d755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:50.825279 11345 consensus_queue.cc:1048] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:50.825387 11353 consensus_queue.cc:1048] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:50.825964 11343 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: TxnStatusTablet state changed. Reason: New leader 864df332d01b445baaa4dd7cfa37d755. Latest consensus state: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.825963 11341 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: TxnStatusTablet state changed. Reason: New leader 864df332d01b445baaa4dd7cfa37d755. Latest consensus state: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.826062 11343 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:50.826069 11341 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:50.827072 11340 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.827174 11340 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:11:50.827234 11340 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.827270 11340 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P 864df332d01b445baaa4dd7cfa37d755: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:11:50.827399 11341 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.827478 11341 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P c47b5f1d6b214c07b98d022bd11bed7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:50.828210 11343 tablet_replica.cc:442] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "864df332d01b445baaa4dd7cfa37d755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17482232f614f468266b744a03b19e0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 40723 } } peers { permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 } } peers { permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 } } }
I20251023 20:11:50.828305 11343 tablet_replica.cc:445] T eda93ccb97bf4f15be1cfe9a72b9ac88 P a17482232f614f468266b744a03b19e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:50.831104 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.829037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.9.117.193:40723)
1023 20:11:50.829086 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.829089 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.829098 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:11:50.829180 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.829181 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.829204 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.829263 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:50.829529 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.830003 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.830566 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.830708 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.830745 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:50.830862 (+ 117us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:50.830937 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.830998 (+ 61us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.831024 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:11:50.836127 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.833780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:35987 (local address 127.9.117.193:42161)
1023 20:11:50.833982 (+ 202us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.833991 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.834062 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.834180 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.834180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.834182 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:11:50.834364 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.834365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.835217 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.835218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.835634 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.835635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.835700 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.835715 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:11:50.835993 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:11:50.835994 (+ 1us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.836026 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.836060 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20251023 20:11:50.836154 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.833853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:42161 (local address 127.9.117.195:35987)
1023 20:11:50.833909 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.833911 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.834069 (+ 158us) server_negotiation.cc:408] Connection header received
1023 20:11:50.834103 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.834104 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.834117 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.834178 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:50.834425 (+ 247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.835148 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.835704 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.835879 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.835901 (+ 22us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:50.835930 (+ 29us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:50.835988 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.836057 (+ 69us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.836082 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:11:50.838904 11124 raft_consensus.cc:1275] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Refusing update from remote peer a17482232f614f468266b744a03b19e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:50.838945 11266 raft_consensus.cc:1275] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Refusing update from remote peer a17482232f614f468266b744a03b19e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:50.839372 11366 consensus_queue.cc:1048] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47b5f1d6b214c07b98d022bd11bed7d" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:50.839547 11343 consensus_queue.cc:1048] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "864df332d01b445baaa4dd7cfa37d755" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:50.848923 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.846110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:40723 (local address 127.9.117.195:44421)
1023 20:11:50.846294 (+ 184us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:50.846307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:50.846359 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:50.846506 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:50.846506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:50.846508 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:11:50.846699 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.846700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.847682 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.847682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:50.848289 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:50.848292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.848393 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.848409 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:11:50.848771 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:11:50.848774 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:11:50.848819 (+ 45us) client_negotiation.cc:240] Negotiation successful
1023 20:11:50.848857 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20251023 20:11:50.848937 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.846310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:44421 (local address 127.9.117.193:40723)
1023 20:11:50.846352 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.846354 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.846363 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:11:50.846416 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.846417 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.846428 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.846505 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:50.846764 (+ 259us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.847577 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.848412 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.848612 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.848638 (+ 26us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:50.848669 (+ 31us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:50.848763 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.848836 (+ 73us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.848864 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251023 20:11:50.859862 10822 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45280:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:50.859998 10822 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45280:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:50.860826 10822 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 5797e6225286431294f39098a20543ec: Sending DeleteTablet for 3 replicas of tablet e057cd796db9434fb64d9ef7bc6467a6
I20251023 20:11:50.861111 10962 tablet_service.cc:1552] Processing DeleteTablet for tablet e057cd796db9434fb64d9ef7bc6467a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:50 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:49064
I20251023 20:11:50.861219 11104 tablet_service.cc:1552] Processing DeleteTablet for tablet e057cd796db9434fb64d9ef7bc6467a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:50 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:41834
I20251023 20:11:50.861340 11359 tablet_replica.cc:333] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: stopping tablet replica
I20251023 20:11:50.861403 11359 raft_consensus.cc:2243] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:11:50.861495 11359 raft_consensus.cc:2272] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:50.861588 11360 tablet_replica.cc:333] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: stopping tablet replica
I20251023 20:11:50.861647 11360 raft_consensus.cc:2243] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:50.861692 11360 raft_consensus.cc:2272] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:50.862098 11360 ts_tablet_manager.cc:1916] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:50.862156 11359 ts_tablet_manager.cc:1916] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:50.863822 11360 ts_tablet_manager.cc:1929] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:50.863876 11360 log.cc:1199] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/e057cd796db9434fb64d9ef7bc6467a6
I20251023 20:11:50.863869 11359 ts_tablet_manager.cc:1929] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:50.863924 11359 log.cc:1199] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/e057cd796db9434fb64d9ef7bc6467a6
I20251023 20:11:50.864104 11360 ts_tablet_manager.cc:1950] T e057cd796db9434fb64d9ef7bc6467a6 P c47b5f1d6b214c07b98d022bd11bed7d: Deleting consensus metadata
I20251023 20:11:50.864146 11359 ts_tablet_manager.cc:1950] T e057cd796db9434fb64d9ef7bc6467a6 P a17482232f614f468266b744a03b19e0: Deleting consensus metadata
I20251023 20:11:50.864473 11246 tablet_service.cc:1552] Processing DeleteTablet for tablet e057cd796db9434fb64d9ef7bc6467a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:50 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:51150
I20251023 20:11:50.864600 11358 tablet_replica.cc:333] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: stopping tablet replica
I20251023 20:11:50.864666 11358 raft_consensus.cc:2243] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:50.864714 11358 raft_consensus.cc:2272] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:50.864693 10808 catalog_manager.cc:4985] TS a17482232f614f468266b744a03b19e0 (127.9.117.193:40723): tablet e057cd796db9434fb64d9ef7bc6467a6 (table test-table [id=af60dddb34a04d28ba15ecfef7e5c67d]) successfully deleted
I20251023 20:11:50.864804 10808 catalog_manager.cc:4985] TS c47b5f1d6b214c07b98d022bd11bed7d (127.9.117.194:35463): tablet e057cd796db9434fb64d9ef7bc6467a6 (table test-table [id=af60dddb34a04d28ba15ecfef7e5c67d]) successfully deleted
I20251023 20:11:50.865742 11358 ts_tablet_manager.cc:1916] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:50.867434 11358 ts_tablet_manager.cc:1929] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:50.867491 11358 log.cc:1199] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/e057cd796db9434fb64d9ef7bc6467a6
I20251023 20:11:50.867722 11358 ts_tablet_manager.cc:1950] T e057cd796db9434fb64d9ef7bc6467a6 P 864df332d01b445baaa4dd7cfa37d755: Deleting consensus metadata
I20251023 20:11:50.868099 10808 catalog_manager.cc:4985] TS 864df332d01b445baaa4dd7cfa37d755 (127.9.117.195:35987): tablet e057cd796db9434fb64d9ef7bc6467a6 (table test-table [id=af60dddb34a04d28ba15ecfef7e5c67d]) successfully deleted
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:11:50.875435 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.865550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.9.117.193:40723)
1023 20:11:50.865618 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.865620 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.865906 (+ 286us) server_negotiation.cc:408] Connection header received
1023 20:11:50.866008 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.866009 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.866019 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.866049 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:50.867064 (+ 1015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.867823 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.868672 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.868871 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.871864 (+ 2993us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:50.871877 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:50.871877 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:50.871886 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:50.874104 (+ 2218us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.874525 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.874529 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.874529 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.874572 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.874745 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.874745 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.874746 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.874872 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:50.875000 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.875223 (+ 223us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.875332 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20251023 20:11:50.875998 11020 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49080
I20251023 20:11:50.884790 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:50.877267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.9.117.254:40487)
1023 20:11:50.877345 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:50.877348 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:50.877450 (+ 102us) server_negotiation.cc:408] Connection header received
1023 20:11:50.877515 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:50.877517 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:50.877534 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:50.877597 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:50.878459 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.879295 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:50.880118 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:50.880311 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:50.881743 (+ 1432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:50.881750 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:50.881751 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:50.881760 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:50.883607 (+ 1847us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.884008 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.884009 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.884010 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.884034 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:50.884187 (+ 153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:50.884188 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:50.884188 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:50.884318 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:50.884425 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:50.884533 (+ 108us) server_negotiation.cc:300] Negotiation successful
1023 20:11:50.884651 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20251023 20:11:50.885142 10822 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45294
I20251023 20:11:50.885917 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 10897
I20251023 20:11:50.892604 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 11039
I20251023 20:11:50.897979 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 11181
I20251023 20:11:50.904142 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 10791
2025-10-23T20:11:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3668 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[11400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[11400](info): set up 2 sockets
Oct 23 20:11:50 dist-test-slave-x62p krb5kdc[11400](info): commencing operation
krb5kdc: starting...
W20251023 20:11:52.948933 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:11:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:11:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:53.096230 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40425
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:40425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:53.178140 11416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:53.178350 11416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:53.178375 11416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:53.179839 11416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:11:53.179884 11416 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251023 20:11:53.179898 11416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:53.179910 11416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:53.179922 11416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:11:53.179934 11416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:11:53.181561 11416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:40425
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:53.181769 11416 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:53.181942 11416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:53.184633 11422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:53.184638 11421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:53.184692 11424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:53.185194 11416 server_base.cc:1047] running on GCE node
I20251023 20:11:53.185415 11416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:53.185688 11416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:53.186843 11416 hybrid_clock.cc:648] HybridClock initialized: now 1761250313186823 us; error 34 us; skew 500 ppm
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:53.189487 11416 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:11:53.190099 11416 webserver.cc:492] Webserver started at http://127.9.117.254:43299/ using document root <none> and password file <none>
I20251023 20:11:53.190287 11416 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:53.190327 11416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:53.190449 11416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:53.191427 11416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "b7f9ce12eadb432e97dc462e7add49c7"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "5f15e18fc5f55bc99715701db929a030"
server_key_iv: "cb1171f6be1b582dac3c1e19ca790ab9"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.191766 11416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "b7f9ce12eadb432e97dc462e7add49c7"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "5f15e18fc5f55bc99715701db929a030"
server_key_iv: "cb1171f6be1b582dac3c1e19ca790ab9"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.193125 11416 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:11:53.193883 11431 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:53.194048 11416 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:11:53.194130 11416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "b7f9ce12eadb432e97dc462e7add49c7"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "5f15e18fc5f55bc99715701db929a030"
server_key_iv: "cb1171f6be1b582dac3c1e19ca790ab9"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.194200 11416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:53.210134 11416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:53.219208 11416 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:53.219362 11416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:53.224812 11416 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:40425
I20251023 20:11:53.224869 11493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:40425 every 8 connection(s)
I20251023 20:11:53.225224 11416 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:11:53.225878 11494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:53.228677 11494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: Bootstrap starting.
I20251023 20:11:53.229409 11494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:53.229689 11494 log.cc:826] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:53.230516 11494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: No bootstrap required, opened a new log
I20251023 20:11:53.231257 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 11416
I20251023 20:11:53.231385 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:11:53.231556 9687 external_mini_cluster.cc:1468] Setting key 753fcba5efdf71e3bd3f5a3793038a1a
I20251023 20:11:53.232197 11494 raft_consensus.cc:359] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f9ce12eadb432e97dc462e7add49c7" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40425 } }
I20251023 20:11:53.232362 11494 raft_consensus.cc:385] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:53.232388 11494 raft_consensus.cc:740] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f9ce12eadb432e97dc462e7add49c7, State: Initialized, Role: FOLLOWER
I20251023 20:11:53.232488 11494 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f9ce12eadb432e97dc462e7add49c7" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40425 } }
I20251023 20:11:53.232554 11494 raft_consensus.cc:399] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:11:53.232580 11494 raft_consensus.cc:493] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:11:53.232618 11494 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:53.233290 11494 raft_consensus.cc:515] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f9ce12eadb432e97dc462e7add49c7" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40425 } }
I20251023 20:11:53.233407 11494 leader_election.cc:304] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7f9ce12eadb432e97dc462e7add49c7; no voters:
I20251023 20:11:53.233572 11494 leader_election.cc:290] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:11:53.233628 11499 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:53.233827 11499 raft_consensus.cc:697] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [term 1 LEADER]: Becoming Leader. State: Replica: b7f9ce12eadb432e97dc462e7add49c7, State: Running, Role: LEADER
I20251023 20:11:53.233872 11494 sys_catalog.cc:565] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:11:53.233984 11499 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f9ce12eadb432e97dc462e7add49c7" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40425 } }
I20251023 20:11:53.234313 11501 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7f9ce12eadb432e97dc462e7add49c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f9ce12eadb432e97dc462e7add49c7" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40425 } } }
I20251023 20:11:53.234378 11501 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [sys.catalog]: This master's current role is: LEADER
I20251023 20:11:53.234597 11504 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:11:53.234916 11500 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7f9ce12eadb432e97dc462e7add49c7. Latest consensus state: current_term: 1 leader_uuid: "b7f9ce12eadb432e97dc462e7add49c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f9ce12eadb432e97dc462e7add49c7" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40425 } } }
I20251023 20:11:53.235051 11500 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7 [sys.catalog]: This master's current role is: LEADER
I20251023 20:11:53.235073 11504 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:11:53.237135 11504 catalog_manager.cc:1357] Generated new cluster ID: 4098d61a89534fadba3e7cd96c269662
I20251023 20:11:53.237233 11504 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:53.241787 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.232152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.9.117.254:40425)
1023 20:11:53.232349 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.232367 (+ 18us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.232393 (+ 26us) server_negotiation.cc:408] Connection header received
1023 20:11:53.232523 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.232530 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.232584 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.232749 (+ 165us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:53.233534 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.235225 (+ 1691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.236192 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.236458 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.238591 (+ 2133us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:53.238613 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:53.238613 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:53.238631 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:53.240381 (+ 1750us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.240757 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.240761 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.240761 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.240819 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.240976 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.240977 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.240978 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.241193 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:53.241350 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.241473 (+ 123us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.241623 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20251023 20:11:53.262336 11504 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:11:53.262954 11504 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:11:53.268226 11504 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: Generated new TSK 0
I20251023 20:11:53.268463 11504 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:53.327448 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40425
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:53.439801 11522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:53.440014 11522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:53.440059 11522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:53.441648 11522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:53.441705 11522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:53.441747 11522 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251023 20:11:53.441766 11522 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:11:53.441797 11522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:11:53.443552 11522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:53.443845 11522 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:53.444047 11522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:53.446899 11530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:53.446931 11522 server_base.cc:1047] running on GCE node
W20251023 20:11:53.446946 11527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:53.446902 11528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:53.447314 11522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:53.447566 11522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:53.448750 11522 hybrid_clock.cc:648] HybridClock initialized: now 1761250313448705 us; error 32 us; skew 500 ppm
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:53.451321 11522 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:11:53.451893 11522 webserver.cc:492] Webserver started at http://127.9.117.193:35779/ using document root <none> and password file <none>
I20251023 20:11:53.452090 11522 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:53.452132 11522 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:53.452250 11522 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:53.453330 11522 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "3e658244616547cfb168bd8d3875a8ff"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "df4dd14bd0bf318fe8fa08c05bb82e95"
server_key_iv: "e49128c142aaa362f8a5125a4c5d9d0a"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.453769 11522 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "3e658244616547cfb168bd8d3875a8ff"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "df4dd14bd0bf318fe8fa08c05bb82e95"
server_key_iv: "e49128c142aaa362f8a5125a4c5d9d0a"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.455041 11522 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:11:53.455753 11537 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:53.455911 11522 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:11:53.456015 11522 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "3e658244616547cfb168bd8d3875a8ff"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "df4dd14bd0bf318fe8fa08c05bb82e95"
server_key_iv: "e49128c142aaa362f8a5125a4c5d9d0a"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.456197 11522 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:53.464322 11522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:53.466382 11522 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:53.466512 11522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:53.467036 11522 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:53.467095 11522 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:53.467139 11522 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:53.467185 11522 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:53.474148 11522 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:34245
I20251023 20:11:53.474210 11652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:34245 every 8 connection(s)
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:53.474577 11522 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:11:53.477980 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.467440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:38625 (local address 127.9.117.254:40425)
1023 20:11:53.467551 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.467554 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.467761 (+ 207us) server_negotiation.cc:408] Connection header received
1023 20:11:53.468298 (+ 537us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.468300 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.468319 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.468405 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:53.469562 (+ 1157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.470456 (+ 894us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.471325 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.471566 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.475094 (+ 3528us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:53.475102 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:53.475102 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:53.475113 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:53.476694 (+ 1581us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.477165 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.477166 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.477167 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.477191 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.477364 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.477365 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.477365 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.477477 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:53.477531 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.477803 (+ 272us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.477901 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251023 20:11:53.478093 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.467408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40425 (local address 127.9.117.193:38625)
1023 20:11:53.467573 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.467615 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.468205 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.468498 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.468502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.469003 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:53.469442 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.469447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.470560 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.470562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.471231 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.471236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.471403 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.472215 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:53.472253 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:53.474933 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:53.476853 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.476861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.476863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.477094 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.477240 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.477241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.477241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.477300 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.477523 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:53.477526 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:53.477673 (+ 147us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.477768 (+ 95us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.477894 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20251023 20:11:53.478655 11653 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40425
I20251023 20:11:53.478760 11653 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:53.479030 11653 heartbeater.cc:507] Master 127.9.117.254:40425 requested a full tablet report, sending...
I20251023 20:11:53.479601 11447 ts_manager.cc:194] Registered new tserver with Master: 3e658244616547cfb168bd8d3875a8ff (127.9.117.193:34245)
I20251023 20:11:53.480625 11447 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:38625
I20251023 20:11:53.483359 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 11522
I20251023 20:11:53.483475 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:11:53.483647 9687 external_mini_cluster.cc:1468] Setting key f567fb61fa951ba5c2d022ea719204bf
I20251023 20:11:53.485244 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.479914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:57765 (local address 127.9.117.254:40425)
1023 20:11:53.479980 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.479982 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.480084 (+ 102us) server_negotiation.cc:408] Connection header received
1023 20:11:53.480149 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.480150 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.480158 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.480208 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:53.480704 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.481271 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.481819 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.482009 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.482710 (+ 701us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:53.482713 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:53.482713 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:53.482721 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:53.484171 (+ 1450us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.484512 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.484514 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.484514 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.484569 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.484793 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.484793 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.484794 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.484887 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:53.484942 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.485102 (+ 160us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.485179 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251023 20:11:53.485378 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.479853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40425 (local address 127.9.117.193:57765)
1023 20:11:53.480009 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.480022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.480094 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.480224 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.480225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.480399 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:53.480630 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.480633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.481351 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.481352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.481740 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.481742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.481823 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.482352 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:53.482356 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:53.482623 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:53.484251 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.484252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.484252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.484430 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.484639 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.484639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.484640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.484677 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.484948 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:53.484949 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:53.485003 (+ 54us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.485082 (+ 79us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.485218 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:53.531951 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40425
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:53.616274 11664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:53.616490 11664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:53.616534 11664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:53.618224 11664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:53.618294 11664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:53.618332 11664 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251023 20:11:53.618350 11664 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:11:53.618381 11664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:11:53.620201 11664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:53.620499 11664 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:53.620718 11664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:53.623620 11669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:53.623654 11670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:53.623773 11672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:53.623963 11664 server_base.cc:1047] running on GCE node
I20251023 20:11:53.624161 11664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:53.624393 11664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:53.625577 11664 hybrid_clock.cc:648] HybridClock initialized: now 1761250313625519 us; error 72 us; skew 500 ppm
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:53.628436 11664 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:11:53.629170 11664 webserver.cc:492] Webserver started at http://127.9.117.194:43115/ using document root <none> and password file <none>
I20251023 20:11:53.629383 11664 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:53.629426 11664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:53.629562 11664 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:53.630597 11664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "4bc6a61bd8af495e9153879b96da40e3"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "e7b346c8ab281a7f20b8499c9dceacc0"
server_key_iv: "b6a6f28b44009816f90d426c87697112"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.630950 11664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "4bc6a61bd8af495e9153879b96da40e3"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "e7b346c8ab281a7f20b8499c9dceacc0"
server_key_iv: "b6a6f28b44009816f90d426c87697112"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.632300 11664 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:11:53.633116 11679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:53.633333 11664 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:11:53.633420 11664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "4bc6a61bd8af495e9153879b96da40e3"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "e7b346c8ab281a7f20b8499c9dceacc0"
server_key_iv: "b6a6f28b44009816f90d426c87697112"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.633500 11664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:53.646504 11664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:53.648804 11664 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:53.648973 11664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:53.649650 11664 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:53.649716 11664 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:53.649752 11664 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:53.649766 11664 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:53.658974 11664 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:35723
I20251023 20:11:53.659032 11794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:35723 every 8 connection(s)
I20251023 20:11:53.659420 11664 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:11:53.660787 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.650124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:48229 (local address 127.9.117.254:40425)
1023 20:11:53.650200 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.650202 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.650455 (+ 253us) server_negotiation.cc:408] Connection header received
1023 20:11:53.651387 (+ 932us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.651389 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.651405 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.651470 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:53.653060 (+ 1590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.653641 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.654279 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.654456 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.657242 (+ 2786us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:53.657247 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:53.657247 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:53.657256 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:53.659198 (+ 1942us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.659814 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.659816 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.659817 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.659852 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.660092 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.660093 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.660093 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.660196 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:53.660249 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.660572 (+ 323us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.660669 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:11:53.660965 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.650092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40425 (local address 127.9.117.194:48229)
1023 20:11:53.650280 (+ 188us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.650316 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.651221 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.651584 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.651590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.652322 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:53.652894 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.652901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.653744 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.653745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.654174 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.654176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.654348 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.655098 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:53.655134 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:53.657088 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:53.659358 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.659364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.659366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.659710 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.659913 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.659914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.659915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.660014 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.660247 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:53.660250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:53.660387 (+ 137us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.660517 (+ 130us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.660651 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20251023 20:11:53.661561 11795 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40425
I20251023 20:11:53.661705 11795 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:53.661924 11795 heartbeater.cc:507] Master 127.9.117.254:40425 requested a full tablet report, sending...
I20251023 20:11:53.662304 11448 ts_manager.cc:194] Registered new tserver with Master: 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723)
I20251023 20:11:53.662837 11448 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:48229
I20251023 20:11:53.667184 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 11664
I20251023 20:11:53.667302 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:11:53.667503 9687 external_mini_cluster.cc:1468] Setting key cd996ce2810230550a9263b6b7e486ea
I20251023 20:11:53.668079 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.662682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:45207 (local address 127.9.117.254:40425)
1023 20:11:53.662760 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.662762 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.662943 (+ 181us) server_negotiation.cc:408] Connection header received
1023 20:11:53.662969 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.662970 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.662981 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.663031 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:53.663551 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.664058 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.664627 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.664760 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.665534 (+ 774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:53.665538 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:53.665538 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:53.665545 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:53.666913 (+ 1368us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.667254 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.667255 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.667255 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.667281 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.667512 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.667513 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.667513 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.667668 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:53.667736 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.667895 (+ 159us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.667985 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251023 20:11:53.668097 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.662624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40425 (local address 127.9.117.194:45207)
1023 20:11:53.662828 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.662838 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.662914 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.663048 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.663049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.663246 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:53.663438 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.663441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.664153 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.664155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.664538 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.664540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.664626 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.665167 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:53.665171 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:53.665446 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:53.666995 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.666996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.666996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.667187 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.667407 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.667408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.667409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.667449 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.667735 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:53.667736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:53.667800 (+ 64us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.667864 (+ 64us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.667987 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:11:53.723778 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40425
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:11:53.811568 11806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:11:53.811789 11806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:11:53.811834 11806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:11:53.813391 11806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:11:53.813447 11806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:11:53.813496 11806 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251023 20:11:53.813517 11806 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:11:53.813558 11806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:11:53.815261 11806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:11:53.815549 11806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:11:53.815754 11806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:11:53.818782 11811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:11:53.818787 11814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:53.819051 11806 server_base.cc:1047] running on GCE node
W20251023 20:11:53.818783 11812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:11:53.819314 11806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:11:53.819550 11806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:11:53.820784 11806 hybrid_clock.cc:648] HybridClock initialized: now 1761250313820762 us; error 36 us; skew 500 ppm
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:11:53.823580 11806 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:11:53.824244 11806 webserver.cc:492] Webserver started at http://127.9.117.195:37451/ using document root <none> and password file <none>
I20251023 20:11:53.824450 11806 fs_manager.cc:362] Metadata directory not provided
I20251023 20:11:53.824494 11806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:11:53.824618 11806 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:11:53.825699 11806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "b2d9714c3e0346f080fd85e11e2a2602"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "830c94d4dfe532608a4922ccc1b86081"
server_key_iv: "e650ba138a126989e96a5b05f4b82aa3"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.826041 11806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "b2d9714c3e0346f080fd85e11e2a2602"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "830c94d4dfe532608a4922ccc1b86081"
server_key_iv: "e650ba138a126989e96a5b05f4b82aa3"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.827432 11806 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:11:53.828279 11821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:53.828558 11806 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:53.828635 11806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "b2d9714c3e0346f080fd85e11e2a2602"
format_stamp: "Formatted at 2025-10-23 20:11:53 on dist-test-slave-x62p"
server_key: "830c94d4dfe532608a4922ccc1b86081"
server_key_iv: "e650ba138a126989e96a5b05f4b82aa3"
server_key_version: "encryptionkey@0"
I20251023 20:11:53.828718 11806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:11:53.840492 11806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:11:53.842757 11806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:11:53.842908 11806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:11:53.843663 11806 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:11:53.843720 11806 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:53.843757 11806 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:11:53.843787 11806 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:53.851347 11806 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:36981
I20251023 20:11:53.851434 11936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:36981 every 8 connection(s)
I20251023 20:11:53.851847 11806 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:11:53.853924 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.844031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:47037 (local address 127.9.117.254:40425)
1023 20:11:53.844106 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.844109 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.844306 (+ 197us) server_negotiation.cc:408] Connection header received
1023 20:11:53.844924 (+ 618us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.844926 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.844943 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.845031 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:53.845967 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.846802 (+ 835us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.847748 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.847975 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.850904 (+ 2929us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:53.850909 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:53.850909 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:53.850917 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:53.852641 (+ 1724us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.853122 (+ 481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.853124 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.853124 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.853156 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.853325 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.853326 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.853326 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.853438 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:53.853512 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.853769 (+ 257us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.853855 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:11:53.854033 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.844012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40425 (local address 127.9.117.195:47037)
1023 20:11:53.844182 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.844216 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.844823 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.845088 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.845092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.845428 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:53.845857 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.845862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.846916 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.846918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.847632 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.847636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.847837 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.848615 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:53.848656 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:53.850763 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:53.852760 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.852768 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.852770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.852997 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.853222 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.853223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.853223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.853277 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.853479 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:53.853482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:53.853614 (+ 132us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.853705 (+ 91us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.853846 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20251023 20:11:53.854584 11937 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40425
I20251023 20:11:53.854676 11937 heartbeater.cc:461] Registering TS with master...
I20251023 20:11:53.854892 11937 heartbeater.cc:507] Master 127.9.117.254:40425 requested a full tablet report, sending...
I20251023 20:11:53.855281 11447 ts_manager.cc:194] Registered new tserver with Master: b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981)
I20251023 20:11:53.855821 11447 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:47037
I20251023 20:11:53.859046 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 11806
I20251023 20:11:53.859133 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:11:53.859339 9687 external_mini_cluster.cc:1468] Setting key a926befef5cf184aa06308e6eb924aab
I20251023 20:11:53.860321 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:11:53.861647 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.855579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40425 (local address 127.9.117.195:45347)
1023 20:11:53.855741 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.855751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.855818 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.855987 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.855988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.856138 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:53.856368 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.856371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.857186 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.857188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.857694 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.857697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.857775 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.858331 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:53.858336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:53.858615 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:53.860723 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.860725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.860725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.860938 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.861113 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:53.861113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:53.861114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:53.861140 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:53.861373 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:53.861373 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:53.861414 (+ 41us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.861460 (+ 46us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.861559 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20251023 20:11:53.861726 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.855648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:45347 (local address 127.9.117.254:40425)
1023 20:11:53.855687 (+ 39us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.855688 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.855824 (+ 136us) server_negotiation.cc:408] Connection header received
1023 20:11:53.855856 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.855857 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.855869 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.855927 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:53.856447 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.857071 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.857810 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.857947 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.858721 (+ 774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:53.858725 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:53.858725 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:53.858732 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:53.860621 (+ 1889us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.861018 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.861019 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.861019 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.861049 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.861188 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.861189 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.861190 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.861318 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:53.861381 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.861495 (+ 114us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.861590 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 23 20:11:53 dist-test-slave-x62p krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:53.883230 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.876412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.9.117.254:40425)
1023 20:11:53.876470 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.876472 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.876480 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:11:53.876508 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.876509 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.876518 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.876582 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:53.877380 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.877892 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.878500 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.878630 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.880880 (+ 2250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:53.880885 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:53.880885 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:53.880893 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:53.882306 (+ 1413us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.882666 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.882667 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.882667 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.882696 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:53.882826 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:53.882827 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:53.882827 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:53.882927 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:53.883011 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.883085 (+ 74us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.883157 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251023 20:11:53.884776 11447 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:11:53.886109 11447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:53.891494 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.888449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34245 (local address 127.0.0.1:48942)
1023 20:11:53.888605 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.888629 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.888709 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.889262 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.889263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.889267 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:53.889501 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.889504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.890553 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.890554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.891199 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.891201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.891275 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.891322 (+ 47us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.891371 (+ 49us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.891411 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":46,"threads_started":1}
I20251023 20:11:53.892225 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.888448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:36981 (local address 127.0.0.1:47230)
1023 20:11:53.888756 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.888766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.888816 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.889130 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.889133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.889135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:53.889385 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.889389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.890952 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.890954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.892037 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.892040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.892087 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.892090 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.892114 (+ 24us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.892157 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20251023 20:11:53.892269 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.888789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.9.117.193:34245)
1023 20:11:53.888972 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.888979 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.888987 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:11:53.889032 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.889034 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.889051 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.889125 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:53.889624 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.890452 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.891310 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.892047 (+ 737us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.892117 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.892149 (+ 32us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.892191 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
I20251023 20:11:53.892937 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.889074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.9.117.194:35723)
1023 20:11:53.889266 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.889274 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.889285 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:11:53.889348 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.889351 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.889367 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.889438 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:53.889895 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.891026 (+ 1131us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.892183 (+ 1157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.892681 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.892772 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.892805 (+ 33us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.892857 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
I20251023 20:11:53.892969 11587 tablet_service.cc:1505] Processing CreateTablet for tablet c5087e124634453a899f157c083d5a53 (DEFAULT_TABLE table=test-table [id=4ebb40a5c9624b8d9053f18cda16f633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:53.893304 11587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5087e124634453a899f157c083d5a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:53.893453 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.888657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.9.117.195:36981)
1023 20:11:53.888911 (+ 254us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:53.888920 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:53.888932 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:11:53.888977 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:53.888980 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:53.888998 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:53.889479 (+ 481us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:53.889699 (+ 220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.890848 (+ 1149us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.892810 (+ 1962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.893281 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.893315 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:53.893339 (+ 24us) server_negotiation.cc:300] Negotiation successful
1023 20:11:53.893369 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":138,"threads_started":1}
I20251023 20:11:53.894038 11871 tablet_service.cc:1505] Processing CreateTablet for tablet c5087e124634453a899f157c083d5a53 (DEFAULT_TABLE table=test-table [id=4ebb40a5c9624b8d9053f18cda16f633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:53.894452 11871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5087e124634453a899f157c083d5a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:53.892344 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:53.888569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:35723 (local address 127.0.0.1:58800)
1023 20:11:53.889063 (+ 494us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:53.889074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:53.889130 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:53.889570 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:53.889570 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:53.889573 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:53.889788 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.889789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.891126 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:53.891127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:53.892080 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:53.892082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:53.892142 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:53.892146 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:11:53.892180 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:11:53.892221 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":107,"threads_started":1}
I20251023 20:11:53.895884 11960 tablet_bootstrap.cc:492] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Bootstrap starting.
I20251023 20:11:53.896706 11960 tablet_bootstrap.cc:654] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:53.896785 11729 tablet_service.cc:1505] Processing CreateTablet for tablet c5087e124634453a899f157c083d5a53 (DEFAULT_TABLE table=test-table [id=4ebb40a5c9624b8d9053f18cda16f633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:53.897109 11729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5087e124634453a899f157c083d5a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:53.897114 11961 tablet_bootstrap.cc:492] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Bootstrap starting.
I20251023 20:11:53.897428 11960 log.cc:826] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:53.897679 11961 tablet_bootstrap.cc:654] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:53.898327 11961 log.cc:826] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:53.898360 11960 tablet_bootstrap.cc:492] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: No bootstrap required, opened a new log
I20251023 20:11:53.898422 11960 ts_tablet_manager.cc:1403] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:11:53.898931 11961 tablet_bootstrap.cc:492] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: No bootstrap required, opened a new log
I20251023 20:11:53.898973 11961 ts_tablet_manager.cc:1403] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:11:53.900187 11960 raft_consensus.cc:359] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:53.900228 11961 raft_consensus.cc:359] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:53.900403 11961 raft_consensus.cc:385] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:53.900014 11964 tablet_bootstrap.cc:492] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Bootstrap starting.
I20251023 20:11:53.900419 11960 raft_consensus.cc:385] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:53.900460 11960 raft_consensus.cc:740] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Initialized, Role: FOLLOWER
I20251023 20:11:53.900485 11961 raft_consensus.cc:740] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d9714c3e0346f080fd85e11e2a2602, State: Initialized, Role: FOLLOWER
I20251023 20:11:53.900596 11960 consensus_queue.cc:260] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:53.900614 11961 consensus_queue.cc:260] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:53.900906 11960 ts_tablet_manager.cc:1434] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251023 20:11:53.900918 11961 ts_tablet_manager.cc:1434] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:11:53.900977 11937 heartbeater.cc:499] Master 127.9.117.254:40425 was elected leader, sending a full tablet report...
I20251023 20:11:53.901065 11653 heartbeater.cc:499] Master 127.9.117.254:40425 was elected leader, sending a full tablet report...
I20251023 20:11:53.901106 11964 tablet_bootstrap.cc:654] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:53.901798 11964 log.cc:826] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Log is configured to *not* fsync() on all Append() calls
I20251023 20:11:53.902499 11964 tablet_bootstrap.cc:492] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: No bootstrap required, opened a new log
I20251023 20:11:53.902556 11964 ts_tablet_manager.cc:1403] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:11:53.903625 11964 raft_consensus.cc:359] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:53.903748 11964 raft_consensus.cc:385] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:53.903767 11964 raft_consensus.cc:740] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc6a61bd8af495e9153879b96da40e3, State: Initialized, Role: FOLLOWER
I20251023 20:11:53.903846 11964 consensus_queue.cc:260] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:53.904033 11964 ts_tablet_manager.cc:1434] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251023 20:11:53.904094 11795 heartbeater.cc:499] Master 127.9.117.254:40425 was elected leader, sending a full tablet report...
W20251023 20:11:53.909992 11796 tablet.cc:2378] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:11:53.975469 11654 tablet.cc:2378] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:11:54.081131 11965 raft_consensus.cc:493] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:54.081264 11965 raft_consensus.cc:515] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:54.081637 11965 leader_election.cc:290] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981), 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723)
I20251023 20:11:54.084100 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.081858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:36981 (local address 127.9.117.193:50117)
1023 20:11:54.082010 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:54.082022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:54.082115 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:54.082261 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:54.082262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:54.082264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:54.082471 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.082473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.083193 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.083194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:54.083831 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.083833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.083916 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.083920 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:11:54.083952 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:11:54.083994 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":22,"thread_start_us":64,"threads_started":1}
I20251023 20:11:54.084105 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.081858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:35723 (local address 127.9.117.193:56313)
1023 20:11:54.082007 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:54.082020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:54.082101 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:54.082251 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:54.082252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:54.082255 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:54.082447 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.082448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.083178 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.083180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:54.083814 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.083816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.083892 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.083899 (+ 7us) client_negotiation.cc:769] Sending connection context
1023 20:11:54.083935 (+ 36us) client_negotiation.cc:240] Negotiation successful
1023 20:11:54.083981 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20251023 20:11:54.084576 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.081943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:56313 (local address 127.9.117.194:35723)
1023 20:11:54.082010 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:54.082013 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:54.082093 (+ 80us) server_negotiation.cc:408] Connection header received
1023 20:11:54.082168 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:54.082169 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:54.082181 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:54.082238 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:54.082519 (+ 281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.083058 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.083911 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.084443 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.084454 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:54.084480 (+ 26us) server_negotiation.cc:300] Negotiation successful
1023 20:11:54.084513 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251023 20:11:54.084583 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.081951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:50117 (local address 127.9.117.195:36981)
1023 20:11:54.082032 (+ 81us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:54.082035 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:54.082113 (+ 78us) server_negotiation.cc:408] Connection header received
1023 20:11:54.082189 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:54.082189 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:54.082198 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:54.082250 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:54.082581 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.083087 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.083940 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.084443 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.084455 (+ 12us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:54.084480 (+ 25us) server_negotiation.cc:300] Negotiation successful
1023 20:11:54.084522 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251023 20:11:54.084896 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5087e124634453a899f157c083d5a53" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3" is_pre_election: true
I20251023 20:11:54.084913 11891 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5087e124634453a899f157c083d5a53" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d9714c3e0346f080fd85e11e2a2602" is_pre_election: true
I20251023 20:11:54.085044 11891 raft_consensus.cc:2468] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 0.
I20251023 20:11:54.085044 11749 raft_consensus.cc:2468] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 0.
I20251023 20:11:54.085309 11539 leader_election.cc:304] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e658244616547cfb168bd8d3875a8ff, 4bc6a61bd8af495e9153879b96da40e3; no voters:
I20251023 20:11:54.085448 11965 raft_consensus.cc:2804] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:54.085508 11965 raft_consensus.cc:493] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:54.085527 11965 raft_consensus.cc:3060] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.086256 11965 raft_consensus.cc:515] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:54.086406 11965 leader_election.cc:290] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 election: Requested vote from peers b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981), 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723)
I20251023 20:11:54.086597 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5087e124634453a899f157c083d5a53" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3"
I20251023 20:11:54.086678 11749 raft_consensus.cc:3060] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.086741 11891 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5087e124634453a899f157c083d5a53" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d9714c3e0346f080fd85e11e2a2602"
I20251023 20:11:54.086850 11891 raft_consensus.cc:3060] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.087678 11749 raft_consensus.cc:2468] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 1.
I20251023 20:11:54.087777 11891 raft_consensus.cc:2468] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 1.
I20251023 20:11:54.087929 11539 leader_election.cc:304] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e658244616547cfb168bd8d3875a8ff, 4bc6a61bd8af495e9153879b96da40e3; no voters:
I20251023 20:11:54.088055 11965 raft_consensus.cc:2804] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:54.088196 11965 raft_consensus.cc:697] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 1 LEADER]: Becoming Leader. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Running, Role: LEADER
I20251023 20:11:54.088310 11965 consensus_queue.cc:237] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } }
I20251023 20:11:54.089138 11448 catalog_manager.cc:5649] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e658244616547cfb168bd8d3875a8ff (127.9.117.193). New cstate: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } health_report { overall_health: UNKNOWN } } }
W20251023 20:11:54.102504 11938 tablet.cc:2378] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 23 20:11:54 dist-test-slave-x62p krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:54.148767 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.141778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.9.117.254:40425)
1023 20:11:54.141842 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:54.141844 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:54.141900 (+ 56us) server_negotiation.cc:408] Connection header received
1023 20:11:54.141969 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:54.141970 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:54.141979 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:54.142033 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:54.142766 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.143251 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.143815 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.143929 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.146204 (+ 2275us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:54.146210 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:54.146211 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:54.146217 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:54.147662 (+ 1445us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:54.148050 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:54.148051 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:54.148052 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:54.148076 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:54.148211 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:54.148212 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:54.148213 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:54.148316 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:54.148375 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:54.148519 (+ 144us) server_negotiation.cc:300] Negotiation successful
1023 20:11:54.148628 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251023 20:11:54.148929 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.141655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40425 (local address 127.0.0.1:50878)
1023 20:11:54.141816 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:54.141830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:54.141914 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:54.142085 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:54.142086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:54.142481 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:11:54.142681 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.142683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.143320 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.143321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:54.143731 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.143733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.143795 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.144424 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:11:54.144429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:11:54.146106 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:11:54.147762 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:54.147771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:54.147774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:54.147982 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:54.148131 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:11:54.148132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:11:54.148133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:11:54.148161 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:11:54.148370 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:11:54.148377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:11:54.148424 (+ 47us) client_negotiation.cc:769] Sending connection context
1023 20:11:54.148496 (+ 72us) client_negotiation.cc:240] Negotiation successful
1023 20:11:54.148805 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":93,"thread_start_us":78,"threads_started":1}
I20251023 20:11:54.152118 11448 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:50878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251023 20:11:54.152360 11448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:54.154013 11871 tablet_service.cc:1505] Processing CreateTablet for tablet 75e4ef5959e64596928bdcb620463f48 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1dc6774faf394ba6a94a7a2096f69274]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:11:54.153996 11729 tablet_service.cc:1505] Processing CreateTablet for tablet 75e4ef5959e64596928bdcb620463f48 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1dc6774faf394ba6a94a7a2096f69274]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:11:54.154027 11587 tablet_service.cc:1505] Processing CreateTablet for tablet 75e4ef5959e64596928bdcb620463f48 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1dc6774faf394ba6a94a7a2096f69274]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:11:54.154184 11587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e4ef5959e64596928bdcb620463f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:54.154183 11871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e4ef5959e64596928bdcb620463f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:54.154184 11729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e4ef5959e64596928bdcb620463f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:54.155346 11961 tablet_bootstrap.cc:492] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: Bootstrap starting.
I20251023 20:11:54.155745 11960 tablet_bootstrap.cc:492] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: Bootstrap starting.
I20251023 20:11:54.155853 11961 tablet_bootstrap.cc:654] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:54.156162 11960 tablet_bootstrap.cc:654] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:54.156559 11964 tablet_bootstrap.cc:492] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: Bootstrap starting.
I20251023 20:11:54.156677 11961 tablet_bootstrap.cc:492] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: No bootstrap required, opened a new log
I20251023 20:11:54.156790 11961 ts_tablet_manager.cc:1403] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:54.156946 11961 raft_consensus.cc:359] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.157016 11961 raft_consensus.cc:385] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:54.157030 11961 raft_consensus.cc:740] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d9714c3e0346f080fd85e11e2a2602, State: Initialized, Role: FOLLOWER
I20251023 20:11:54.157068 11961 consensus_queue.cc:260] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.157162 11961 ts_tablet_manager.cc:1434] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:54.157229 11966 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.157282 11966 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:54.157284 11964 tablet_bootstrap.cc:654] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:54.157860 11960 tablet_bootstrap.cc:492] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: No bootstrap required, opened a new log
I20251023 20:11:54.157912 11960 ts_tablet_manager.cc:1403] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:11:54.158063 11960 raft_consensus.cc:359] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.158119 11960 raft_consensus.cc:385] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:54.158123 11964 tablet_bootstrap.cc:492] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: No bootstrap required, opened a new log
I20251023 20:11:54.158144 11960 raft_consensus.cc:740] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Initialized, Role: FOLLOWER
I20251023 20:11:54.158164 11964 ts_tablet_manager.cc:1403] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:11:54.158203 11960 consensus_queue.cc:260] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.158313 11960 ts_tablet_manager.cc:1434] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:54.158339 11964 raft_consensus.cc:359] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.158370 11965 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.158430 11964 raft_consensus.cc:385] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:54.158454 11964 raft_consensus.cc:740] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc6a61bd8af495e9153879b96da40e3, State: Initialized, Role: FOLLOWER
I20251023 20:11:54.158442 11965 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:54.158499 11964 consensus_queue.cc:260] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.158602 11964 ts_tablet_manager.cc:1434] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:54.158746 11968 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.158866 11968 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:54.538043 11965 raft_consensus.cc:493] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:54.538163 11965 raft_consensus.cc:515] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.538342 11965 leader_election.cc:290] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723), b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981)
I20251023 20:11:54.538678 11891 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e4ef5959e64596928bdcb620463f48" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d9714c3e0346f080fd85e11e2a2602" is_pre_election: true
I20251023 20:11:54.538748 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e4ef5959e64596928bdcb620463f48" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3" is_pre_election: true
I20251023 20:11:54.538822 11891 raft_consensus.cc:2468] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 0.
I20251023 20:11:54.538829 11749 raft_consensus.cc:2468] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 0.
I20251023 20:11:54.539002 11539 leader_election.cc:304] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e658244616547cfb168bd8d3875a8ff, 4bc6a61bd8af495e9153879b96da40e3; no voters:
I20251023 20:11:54.539129 11965 raft_consensus.cc:2804] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:54.539192 11965 raft_consensus.cc:493] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:54.539209 11965 raft_consensus.cc:3060] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.539973 11965 raft_consensus.cc:515] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.540113 11965 leader_election.cc:290] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 election: Requested vote from peers 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723), b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981)
I20251023 20:11:54.540344 11891 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e4ef5959e64596928bdcb620463f48" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d9714c3e0346f080fd85e11e2a2602"
I20251023 20:11:54.540349 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e4ef5959e64596928bdcb620463f48" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3"
I20251023 20:11:54.540427 11891 raft_consensus.cc:3060] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.540431 11749 raft_consensus.cc:3060] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.541085 11749 raft_consensus.cc:2468] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 1.
I20251023 20:11:54.541282 11891 raft_consensus.cc:2468] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 1.
I20251023 20:11:54.541313 11539 leader_election.cc:304] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e658244616547cfb168bd8d3875a8ff, 4bc6a61bd8af495e9153879b96da40e3; no voters:
I20251023 20:11:54.541406 11965 raft_consensus.cc:2804] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:54.541465 11965 raft_consensus.cc:697] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [term 1 LEADER]: Becoming Leader. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Running, Role: LEADER
I20251023 20:11:54.541517 11965 consensus_queue.cc:237] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.541829 11971 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: TxnStatusTablet state changed. Reason: New leader 3e658244616547cfb168bd8d3875a8ff. Latest consensus state: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.541934 11971 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:11:54.542095 11983 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251023 20:11:54.542214 11983 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251023 20:11:54.542205 11448 catalog_manager.cc:5649] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e658244616547cfb168bd8d3875a8ff (127.9.117.193). New cstate: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } health_report { overall_health: UNKNOWN } } }
I20251023 20:11:54.604830 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.602408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34245 (local address 127.0.0.1:48944)
1023 20:11:54.602491 (+ 83us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:54.602502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:54.602566 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:54.602790 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:54.602791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:54.602792 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:11:54.602970 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.602972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.603709 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.603710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:54.604130 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.604131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.604199 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.604222 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:11:54.604604 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:11:54.604616 (+ 12us) client_negotiation.cc:769] Sending connection context
1023 20:11:54.604669 (+ 53us) client_negotiation.cc:240] Negotiation successful
1023 20:11:54.604760 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251023 20:11:54.604836 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.602505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.9.117.193:34245)
1023 20:11:54.602636 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:54.602638 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:54.602648 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:11:54.602689 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:54.602690 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:54.602708 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:54.602782 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:54.603060 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.603627 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.604207 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.604381 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.604417 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:54.604535 (+ 118us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:54.604611 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:54.604679 (+ 68us) server_negotiation.cc:300] Negotiation successful
1023 20:11:54.604708 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20251023 20:11:54.606613 11749 raft_consensus.cc:1275] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Refusing update from remote peer 3e658244616547cfb168bd8d3875a8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:54.606616 11891 raft_consensus.cc:1275] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Refusing update from remote peer 3e658244616547cfb168bd8d3875a8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:54.606956 11971 consensus_queue.cc:1048] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:54.607092 11965 consensus_queue.cc:1048] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:54.608292 11966 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: TxnStatusTablet state changed. Reason: New leader 3e658244616547cfb168bd8d3875a8ff. Latest consensus state: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.608294 11968 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: TxnStatusTablet state changed. Reason: New leader 3e658244616547cfb168bd8d3875a8ff. Latest consensus state: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.608381 11966 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:54.608382 11968 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:54.609663 11988 mvcc.cc:204] Tried to move back new op lower bound from 7214081288626049024 to 7214081288362532864. Current Snapshot: MvccSnapshot[applied={T|T < 7214081288626049024}]
I20251023 20:11:54.609730 11989 mvcc.cc:204] Tried to move back new op lower bound from 7214081288626049024 to 7214081288362532864. Current Snapshot: MvccSnapshot[applied={T|T < 7214081288626049024}]
I20251023 20:11:54.609747 11968 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.609822 11968 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P 4bc6a61bd8af495e9153879b96da40e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:54.609835 11966 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.609907 11966 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P b2d9714c3e0346f080fd85e11e2a2602: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:11:54.609891 11965 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.609969 11965 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:11:54.610097 11965 tablet_replica.cc:442] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } } }
I20251023 20:11:54.610142 11965 tablet_replica.cc:445] T 75e4ef5959e64596928bdcb620463f48 P 3e658244616547cfb168bd8d3875a8ff: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:11:54.610503 11987 mvcc.cc:204] Tried to move back new op lower bound from 7214081288626049024 to 7214081288362532864. Current Snapshot: MvccSnapshot[applied={T|T < 7214081288626049024 or (T in {7214081288626049024})}]
I20251023 20:11:54.614588 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.612163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.9.117.193:34245)
1023 20:11:54.612208 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:54.612211 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:54.612297 (+ 86us) server_negotiation.cc:408] Connection header received
1023 20:11:54.612359 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:54.612359 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:54.612369 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:54.612416 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:54.612805 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.613334 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.614188 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.614317 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.614334 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:54.614355 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:54.614401 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:54.614472 (+ 71us) server_negotiation.cc:300] Negotiation successful
1023 20:11:54.614490 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:11:54.619609 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.616994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:59831 (local address 127.9.117.193:34245)
1023 20:11:54.617036 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:54.617037 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:54.617152 (+ 115us) server_negotiation.cc:408] Connection header received
1023 20:11:54.617213 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:54.617214 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:54.617222 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:54.617266 (+ 44us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:54.617918 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.618392 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.619241 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.619351 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.619367 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:54.619389 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:54.619433 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:54.619505 (+ 72us) server_negotiation.cc:300] Negotiation successful
1023 20:11:54.619543 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:11:54.619699 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:54.616923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34245 (local address 127.9.117.193:59831)
1023 20:11:54.617078 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:54.617091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:54.617157 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:54.617282 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:54.617283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:54.617285 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:11:54.617830 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.617832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.618483 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:54.618484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:54.619165 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:54.619168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:54.619243 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:54.619260 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:11:54.619450 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:11:54.619452 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:11:54.619501 (+ 49us) client_negotiation.cc:240] Negotiation successful
1023 20:11:54.619549 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20251023 20:11:54.622066 11749 raft_consensus.cc:1275] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Refusing update from remote peer 3e658244616547cfb168bd8d3875a8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:54.622104 11891 raft_consensus.cc:1275] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Refusing update from remote peer 3e658244616547cfb168bd8d3875a8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:54.622275 11992 consensus_queue.cc:1048] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:54.622366 11965 consensus_queue.cc:1048] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:54.634387 11446 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50872:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:54.634516 11446 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50872:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:54.635327 11446 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: Sending DeleteTablet for 3 replicas of tablet c5087e124634453a899f157c083d5a53
I20251023 20:11:54.635679 11729 tablet_service.cc:1552] Processing DeleteTablet for tablet c5087e124634453a899f157c083d5a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:54 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:58800
I20251023 20:11:54.635775 11871 tablet_service.cc:1552] Processing DeleteTablet for tablet c5087e124634453a899f157c083d5a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:54 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:47230
I20251023 20:11:54.636039 12005 tablet_replica.cc:333] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: stopping tablet replica
I20251023 20:11:54.636029 11586 tablet_service.cc:1552] Processing DeleteTablet for tablet c5087e124634453a899f157c083d5a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:54 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:48942
I20251023 20:11:54.636137 12005 raft_consensus.cc:2243] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:54.636202 12005 raft_consensus.cc:2272] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:54.636541 12005 ts_tablet_manager.cc:1916] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:54.636483 11446 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:11:54.636629 11446 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:54.638187 12006 tablet_replica.cc:333] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: stopping tablet replica
I20251023 20:11:54.638288 12006 raft_consensus.cc:2243] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:11:54.638444 12005 ts_tablet_manager.cc:1929] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:54.638573 11729 tablet_service.cc:1505] Processing CreateTablet for tablet 7ab705ace72b4e22922b8b9191620490 (DEFAULT_TABLE table=test-table [id=57e47c6856404220b1b2ef270fe7d003]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:54.638569 11871 tablet_service.cc:1505] Processing CreateTablet for tablet 7ab705ace72b4e22922b8b9191620490 (DEFAULT_TABLE table=test-table [id=57e47c6856404220b1b2ef270fe7d003]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:54.638702 11729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab705ace72b4e22922b8b9191620490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:54.638579 12005 log.cc:1199] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/c5087e124634453a899f157c083d5a53
I20251023 20:11:54.638700 11871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab705ace72b4e22922b8b9191620490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:54.639109 12005 ts_tablet_manager.cc:1950] T c5087e124634453a899f157c083d5a53 P b2d9714c3e0346f080fd85e11e2a2602: Deleting consensus metadata
I20251023 20:11:54.640015 11964 tablet_bootstrap.cc:492] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: Bootstrap starting.
I20251023 20:11:54.638403 12006 raft_consensus.cc:2272] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:54.640146 11961 tablet_bootstrap.cc:492] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: Bootstrap starting.
I20251023 20:11:54.640488 11964 tablet_bootstrap.cc:654] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:54.640554 11961 tablet_bootstrap.cc:654] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:54.640588 12006 ts_tablet_manager.cc:1916] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:54.638842 11586 tablet_service.cc:1505] Processing CreateTablet for tablet 7ab705ace72b4e22922b8b9191620490 (DEFAULT_TABLE table=test-table [id=57e47c6856404220b1b2ef270fe7d003]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:54.641297 11586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab705ace72b4e22922b8b9191620490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:54.642256 11434 catalog_manager.cc:4985] TS b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981): tablet c5087e124634453a899f157c083d5a53 (table test-table [id=4ebb40a5c9624b8d9053f18cda16f633]) successfully deleted
I20251023 20:11:54.642295 12006 ts_tablet_manager.cc:1929] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:54.642446 11964 tablet_bootstrap.cc:492] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: No bootstrap required, opened a new log
I20251023 20:11:54.642503 11964 ts_tablet_manager.cc:1403] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20251023 20:11:54.642529 12006 log.cc:1199] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/c5087e124634453a899f157c083d5a53
I20251023 20:11:54.642665 11964 raft_consensus.cc:359] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.642771 11964 raft_consensus.cc:385] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:54.642802 11964 raft_consensus.cc:740] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc6a61bd8af495e9153879b96da40e3, State: Initialized, Role: FOLLOWER
I20251023 20:11:54.642799 11961 tablet_bootstrap.cc:492] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: No bootstrap required, opened a new log
I20251023 20:11:54.642845 11961 ts_tablet_manager.cc:1403] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:11:54.642848 11964 consensus_queue.cc:260] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.642958 11964 ts_tablet_manager.cc:1434] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:54.643018 11961 raft_consensus.cc:359] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.643095 11961 raft_consensus.cc:385] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:54.643121 11961 raft_consensus.cc:740] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d9714c3e0346f080fd85e11e2a2602, State: Initialized, Role: FOLLOWER
I20251023 20:11:54.643167 11961 consensus_queue.cc:260] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.643260 11961 ts_tablet_manager.cc:1434] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251023 20:11:54.643344 12006 ts_tablet_manager.cc:1950] T c5087e124634453a899f157c083d5a53 P 3e658244616547cfb168bd8d3875a8ff: Deleting consensus metadata
I20251023 20:11:54.644589 11960 tablet_bootstrap.cc:492] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: Bootstrap starting.
I20251023 20:11:54.644876 11432 catalog_manager.cc:4985] TS 3e658244616547cfb168bd8d3875a8ff (127.9.117.193:34245): tablet c5087e124634453a899f157c083d5a53 (table test-table [id=4ebb40a5c9624b8d9053f18cda16f633]) successfully deleted
I20251023 20:11:54.645061 11960 tablet_bootstrap.cc:654] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:54.645149 12004 tablet_replica.cc:333] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: stopping tablet replica
I20251023 20:11:54.645256 12004 raft_consensus.cc:2243] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:54.645323 12004 raft_consensus.cc:2272] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:54.645730 11960 tablet_bootstrap.cc:492] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: No bootstrap required, opened a new log
I20251023 20:11:54.645737 12004 ts_tablet_manager.cc:1916] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:54.645776 11960 ts_tablet_manager.cc:1403] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251023 20:11:54.645936 11960 raft_consensus.cc:359] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.645992 11960 raft_consensus.cc:385] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:54.646016 11960 raft_consensus.cc:740] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Initialized, Role: FOLLOWER
I20251023 20:11:54.646070 11960 consensus_queue.cc:260] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.646165 11960 ts_tablet_manager.cc:1434] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:54.647990 12004 ts_tablet_manager.cc:1929] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:54.648083 12004 log.cc:1199] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/c5087e124634453a899f157c083d5a53
I20251023 20:11:54.648367 12004 ts_tablet_manager.cc:1950] T c5087e124634453a899f157c083d5a53 P 4bc6a61bd8af495e9153879b96da40e3: Deleting consensus metadata
I20251023 20:11:54.648841 11433 catalog_manager.cc:4985] TS 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723): tablet c5087e124634453a899f157c083d5a53 (table test-table [id=4ebb40a5c9624b8d9053f18cda16f633]) successfully deleted
I20251023 20:11:54.875722 11965 raft_consensus.cc:493] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:54.875852 11965 raft_consensus.cc:515] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.876056 11965 leader_election.cc:290] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723), b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981)
I20251023 20:11:54.876295 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ab705ace72b4e22922b8b9191620490" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3" is_pre_election: true
I20251023 20:11:54.876305 11891 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ab705ace72b4e22922b8b9191620490" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d9714c3e0346f080fd85e11e2a2602" is_pre_election: true
I20251023 20:11:54.876389 11749 raft_consensus.cc:2468] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 0.
I20251023 20:11:54.876408 11891 raft_consensus.cc:2468] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 0.
I20251023 20:11:54.876578 11539 leader_election.cc:304] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e658244616547cfb168bd8d3875a8ff, 4bc6a61bd8af495e9153879b96da40e3; no voters:
I20251023 20:11:54.876703 11965 raft_consensus.cc:2804] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:54.876780 11965 raft_consensus.cc:493] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:54.876797 11965 raft_consensus.cc:3060] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.877557 11965 raft_consensus.cc:515] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.877696 11965 leader_election.cc:290] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 election: Requested vote from peers 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723), b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981)
I20251023 20:11:54.877853 11891 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ab705ace72b4e22922b8b9191620490" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d9714c3e0346f080fd85e11e2a2602"
I20251023 20:11:54.877918 11891 raft_consensus.cc:3060] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.877893 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ab705ace72b4e22922b8b9191620490" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3"
I20251023 20:11:54.877959 11749 raft_consensus.cc:3060] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:54.878593 11891 raft_consensus.cc:2468] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 1.
I20251023 20:11:54.878610 11749 raft_consensus.cc:2468] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 1.
I20251023 20:11:54.878805 11540 leader_election.cc:304] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e658244616547cfb168bd8d3875a8ff, b2d9714c3e0346f080fd85e11e2a2602; no voters:
I20251023 20:11:54.878933 11965 raft_consensus.cc:2804] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:54.878988 11965 raft_consensus.cc:697] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 1 LEADER]: Becoming Leader. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Running, Role: LEADER
I20251023 20:11:54.879046 11965 consensus_queue.cc:237] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:54.879698 11446 catalog_manager.cc:5649] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e658244616547cfb168bd8d3875a8ff (127.9.117.193). New cstate: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } health_report { overall_health: UNKNOWN } } }
I20251023 20:11:54.931212 11891 raft_consensus.cc:1275] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Refusing update from remote peer 3e658244616547cfb168bd8d3875a8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:54.931254 11749 raft_consensus.cc:1275] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Refusing update from remote peer 3e658244616547cfb168bd8d3875a8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:54.931478 11992 consensus_queue.cc:1048] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:54.931589 11965 consensus_queue.cc:1048] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:11:54.936411 11987 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251023 20:11:56.454850 11448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50872:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:56.454998 11448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50872:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:56.455852 11448 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: Sending DeleteTablet for 3 replicas of tablet 7ab705ace72b4e22922b8b9191620490
I20251023 20:11:56.456113 11586 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ab705ace72b4e22922b8b9191620490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:48942
I20251023 20:11:56.456164 11729 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ab705ace72b4e22922b8b9191620490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:58800
I20251023 20:11:56.456176 11871 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ab705ace72b4e22922b8b9191620490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:47230
I20251023 20:11:56.456387 12034 tablet_replica.cc:333] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: stopping tablet replica
I20251023 20:11:56.456404 12033 tablet_replica.cc:333] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: stopping tablet replica
I20251023 20:11:56.456451 12034 raft_consensus.cc:2243] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:56.456499 12034 raft_consensus.cc:2272] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:56.456540 12033 raft_consensus.cc:2243] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:56.456640 12033 raft_consensus.cc:2272] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:56.456817 12034 ts_tablet_manager.cc:1916] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:56.457023 12033 ts_tablet_manager.cc:1916] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:56.458861 12034 ts_tablet_manager.cc:1929] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251023 20:11:56.458974 12034 log.cc:1199] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/7ab705ace72b4e22922b8b9191620490
I20251023 20:11:56.459125 12033 ts_tablet_manager.cc:1929] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251023 20:11:56.459228 12033 log.cc:1199] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/7ab705ace72b4e22922b8b9191620490
I20251023 20:11:56.459254 12034 ts_tablet_manager.cc:1950] T 7ab705ace72b4e22922b8b9191620490 P b2d9714c3e0346f080fd85e11e2a2602: Deleting consensus metadata
I20251023 20:11:56.459252 12032 tablet_replica.cc:333] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: stopping tablet replica
I20251023 20:11:56.459312 12032 raft_consensus.cc:2243] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:11:56.459420 12032 raft_consensus.cc:2272] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:56.459689 12032 ts_tablet_manager.cc:1916] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:56.459605 12033 ts_tablet_manager.cc:1950] T 7ab705ace72b4e22922b8b9191620490 P 4bc6a61bd8af495e9153879b96da40e3: Deleting consensus metadata
I20251023 20:11:56.459790 11434 catalog_manager.cc:4985] TS b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981): tablet 7ab705ace72b4e22922b8b9191620490 (table test-table [id=57e47c6856404220b1b2ef270fe7d003]) successfully deleted
I20251023 20:11:56.460196 11433 catalog_manager.cc:4985] TS 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723): tablet 7ab705ace72b4e22922b8b9191620490 (table test-table [id=57e47c6856404220b1b2ef270fe7d003]) successfully deleted
I20251023 20:11:56.461270 12032 ts_tablet_manager.cc:1929] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251023 20:11:56.461326 12032 log.cc:1199] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/7ab705ace72b4e22922b8b9191620490
I20251023 20:11:56.461537 12032 ts_tablet_manager.cc:1950] T 7ab705ace72b4e22922b8b9191620490 P 3e658244616547cfb168bd8d3875a8ff: Deleting consensus metadata
I20251023 20:11:56.462167 11432 catalog_manager.cc:4985] TS 3e658244616547cfb168bd8d3875a8ff (127.9.117.193:34245): tablet 7ab705ace72b4e22922b8b9191620490 (table test-table [id=57e47c6856404220b1b2ef270fe7d003]) successfully deleted
Oct 23 20:11:56 dist-test-slave-x62p krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 23 20:11:56 dist-test-slave-x62p krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:11:56.480619 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.473206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.9.117.254:40425)
1023 20:11:56.473354 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:56.473356 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:56.473370 (+ 14us) server_negotiation.cc:408] Connection header received
1023 20:11:56.473465 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:56.473466 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:56.473479 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:56.473564 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:11:56.474293 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.474872 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.475452 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.475663 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.477914 (+ 2251us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:11:56.477919 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:11:56.477919 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:11:56.477927 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:11:56.479568 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:56.479890 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:56.479891 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:56.479892 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:56.479917 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:11:56.480053 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:11:56.480054 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:11:56.480055 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:11:56.480154 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:11:56.480253 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:56.480357 (+ 104us) server_negotiation.cc:300] Negotiation successful
1023 20:11:56.480473 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20251023 20:11:56.482357 11448 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:11:56.482520 11448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:56.484035 11586 tablet_service.cc:1505] Processing CreateTablet for tablet e70c3bdd8f0d43308357dcba66cd2707 (DEFAULT_TABLE table=test-table [id=49ed46dd80064adeab4d0ed079b7a944]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:56.484040 11871 tablet_service.cc:1505] Processing CreateTablet for tablet e70c3bdd8f0d43308357dcba66cd2707 (DEFAULT_TABLE table=test-table [id=49ed46dd80064adeab4d0ed079b7a944]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:56.484032 11729 tablet_service.cc:1505] Processing CreateTablet for tablet e70c3bdd8f0d43308357dcba66cd2707 (DEFAULT_TABLE table=test-table [id=49ed46dd80064adeab4d0ed079b7a944]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:56.484177 11871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70c3bdd8f0d43308357dcba66cd2707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:56.484176 11729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70c3bdd8f0d43308357dcba66cd2707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:56.484179 11586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e70c3bdd8f0d43308357dcba66cd2707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:56.485579 12043 tablet_bootstrap.cc:492] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: Bootstrap starting.
I20251023 20:11:56.485867 12044 tablet_bootstrap.cc:492] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: Bootstrap starting.
I20251023 20:11:56.485946 12045 tablet_bootstrap.cc:492] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: Bootstrap starting.
I20251023 20:11:56.486106 12043 tablet_bootstrap.cc:654] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:56.486320 12044 tablet_bootstrap.cc:654] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:56.486533 12045 tablet_bootstrap.cc:654] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:56.486817 12043 tablet_bootstrap.cc:492] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: No bootstrap required, opened a new log
I20251023 20:11:56.486909 12043 ts_tablet_manager.cc:1403] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:56.487057 12044 tablet_bootstrap.cc:492] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: No bootstrap required, opened a new log
I20251023 20:11:56.487112 12044 ts_tablet_manager.cc:1403] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:56.487098 12043 raft_consensus.cc:359] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.487207 12043 raft_consensus.cc:385] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:56.487249 12045 tablet_bootstrap.cc:492] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: No bootstrap required, opened a new log
I20251023 20:11:56.487262 12043 raft_consensus.cc:740] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Initialized, Role: FOLLOWER
I20251023 20:11:56.487305 12045 ts_tablet_manager.cc:1403] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:56.487318 12044 raft_consensus.cc:359] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.487393 12044 raft_consensus.cc:385] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:56.487357 12043 consensus_queue.cc:260] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.487458 12044 raft_consensus.cc:740] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d9714c3e0346f080fd85e11e2a2602, State: Initialized, Role: FOLLOWER
I20251023 20:11:56.487484 12043 ts_tablet_manager.cc:1434] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:11:56.487465 12045 raft_consensus.cc:359] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.487540 12045 raft_consensus.cc:385] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:56.487543 12044 consensus_queue.cc:260] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.487587 12045 raft_consensus.cc:740] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc6a61bd8af495e9153879b96da40e3, State: Initialized, Role: FOLLOWER
I20251023 20:11:56.487644 12045 consensus_queue.cc:260] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.487743 12044 ts_tablet_manager.cc:1434] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:56.487843 12045 ts_tablet_manager.cc:1434] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251023 20:11:56.610981 12049 raft_consensus.cc:493] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:56.611114 12049 raft_consensus.cc:515] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.611486 12049 leader_election.cc:290] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e658244616547cfb168bd8d3875a8ff (127.9.117.193:34245), 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723)
I20251023 20:11:56.614949 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.611694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:35723 (local address 127.9.117.195:51365)
1023 20:11:56.611915 (+ 221us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:56.611928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:56.611998 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:56.612186 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:56.612186 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:56.612189 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:56.612540 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:56.612542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.613628 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.613630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:56.614655 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:56.614659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.614768 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.614770 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:11:56.614802 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:11:56.614837 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":140,"threads_started":1}
I20251023 20:11:56.614949 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.611695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34245 (local address 127.9.117.195:50071)
1023 20:11:56.611915 (+ 220us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:56.611928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:56.611998 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:56.612185 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:56.612186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:56.612189 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:11:56.612499 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:56.612501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.613628 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.613630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:56.614678 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:56.614679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.614750 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.614756 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:11:56.614792 (+ 36us) client_negotiation.cc:240] Negotiation successful
1023 20:11:56.614837 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":3,"thread_start_us":132,"threads_started":1}
I20251023 20:11:56.615412 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.611774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:50071 (local address 127.9.117.193:34245)
1023 20:11:56.611978 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:56.611980 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:56.611996 (+ 16us) server_negotiation.cc:408] Connection header received
1023 20:11:56.612066 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:56.612067 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:56.612079 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:56.612179 (+ 100us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:56.612630 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.613485 (+ 855us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.614782 (+ 1297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.615272 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.615281 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:56.615308 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:11:56.615340 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20251023 20:11:56.615423 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.611818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:51365 (local address 127.9.117.194:35723)
1023 20:11:56.611975 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:56.611977 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:56.611996 (+ 19us) server_negotiation.cc:408] Connection header received
1023 20:11:56.612066 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:56.612067 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:56.612079 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:56.612171 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:11:56.612651 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.613485 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.614831 (+ 1346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.615280 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.615289 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:56.615311 (+ 22us) server_negotiation.cc:300] Negotiation successful
1023 20:11:56.615340 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20251023 20:11:56.615687 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e70c3bdd8f0d43308357dcba66cd2707" candidate_uuid: "b2d9714c3e0346f080fd85e11e2a2602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3" is_pre_election: true
I20251023 20:11:56.615715 11607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e70c3bdd8f0d43308357dcba66cd2707" candidate_uuid: "b2d9714c3e0346f080fd85e11e2a2602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e658244616547cfb168bd8d3875a8ff" is_pre_election: true
I20251023 20:11:56.615788 11749 raft_consensus.cc:2468] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d9714c3e0346f080fd85e11e2a2602 in term 0.
I20251023 20:11:56.615818 11607 raft_consensus.cc:2468] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d9714c3e0346f080fd85e11e2a2602 in term 0.
I20251023 20:11:56.615975 11823 leader_election.cc:304] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc6a61bd8af495e9153879b96da40e3, b2d9714c3e0346f080fd85e11e2a2602; no voters:
I20251023 20:11:56.616146 12049 raft_consensus.cc:2804] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:56.616186 12049 raft_consensus.cc:493] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:56.616211 12049 raft_consensus.cc:3060] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:56.616916 12049 raft_consensus.cc:515] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.617059 12049 leader_election.cc:290] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [CANDIDATE]: Term 1 election: Requested vote from peers 3e658244616547cfb168bd8d3875a8ff (127.9.117.193:34245), 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723)
I20251023 20:11:56.617233 11607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e70c3bdd8f0d43308357dcba66cd2707" candidate_uuid: "b2d9714c3e0346f080fd85e11e2a2602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e658244616547cfb168bd8d3875a8ff"
I20251023 20:11:56.617245 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e70c3bdd8f0d43308357dcba66cd2707" candidate_uuid: "b2d9714c3e0346f080fd85e11e2a2602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3"
I20251023 20:11:56.617311 11607 raft_consensus.cc:3060] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:56.617313 11749 raft_consensus.cc:3060] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:56.617969 11749 raft_consensus.cc:2468] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d9714c3e0346f080fd85e11e2a2602 in term 1.
I20251023 20:11:56.618053 11607 raft_consensus.cc:2468] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d9714c3e0346f080fd85e11e2a2602 in term 1.
I20251023 20:11:56.618165 11823 leader_election.cc:304] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc6a61bd8af495e9153879b96da40e3, b2d9714c3e0346f080fd85e11e2a2602; no voters:
I20251023 20:11:56.618280 12049 raft_consensus.cc:2804] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:56.618454 12049 raft_consensus.cc:697] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 LEADER]: Becoming Leader. State: Replica: b2d9714c3e0346f080fd85e11e2a2602, State: Running, Role: LEADER
I20251023 20:11:56.618547 12049 consensus_queue.cc:237] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.619308 11447 catalog_manager.cc:5649] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195). New cstate: current_term: 1 leader_uuid: "b2d9714c3e0346f080fd85e11e2a2602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } health_report { overall_health: HEALTHY } } }
I20251023 20:11:56.637645 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.635017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.9.117.195:36981)
1023 20:11:56.635155 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:56.635157 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:56.635166 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:11:56.635204 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:56.635205 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:56.635226 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:56.635288 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:56.635555 (+ 267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.636378 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.637021 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.637178 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.637232 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:56.637355 (+ 123us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:56.637429 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:56.637487 (+ 58us) server_negotiation.cc:300] Negotiation successful
1023 20:11:56.637512 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20251023 20:11:56.641995 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.639826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:59859 (local address 127.9.117.193:34245)
1023 20:11:56.639888 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:56.639890 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:56.639978 (+ 88us) server_negotiation.cc:408] Connection header received
1023 20:11:56.640050 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:56.640051 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:56.640064 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:56.640114 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:56.640499 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.641013 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.641582 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.641740 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.641758 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:56.641782 (+ 24us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:56.641829 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:56.641898 (+ 69us) server_negotiation.cc:300] Negotiation successful
1023 20:11:56.641921 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251023 20:11:56.642014 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.639737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34245 (local address 127.9.117.195:59859)
1023 20:11:56.639908 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:56.639919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:56.639986 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:56.640121 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:56.640122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:56.640123 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:11:56.640394 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:56.640398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.641084 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.641085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:56.641508 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:56.641510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.641577 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.641594 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:11:56.641841 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:11:56.641843 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:11:56.641891 (+ 48us) client_negotiation.cc:240] Negotiation successful
1023 20:11:56.641929 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20251023 20:11:56.644048 11749 raft_consensus.cc:1275] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Refusing update from remote peer b2d9714c3e0346f080fd85e11e2a2602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:56.644131 11607 raft_consensus.cc:1275] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Refusing update from remote peer b2d9714c3e0346f080fd85e11e2a2602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:56.644311 12055 consensus_queue.cc:1048] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:56.644425 12049 consensus_queue.cc:1048] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:56.651513 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.649462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:48787 (local address 127.9.117.195:36981)
1023 20:11:56.649512 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:56.649514 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:56.649539 (+ 25us) server_negotiation.cc:408] Connection header received
1023 20:11:56.649616 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:56.649617 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:56.649631 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:56.649675 (+ 44us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:56.649940 (+ 265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.650443 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.651049 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.651185 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.651202 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:56.651226 (+ 24us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:56.651277 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:56.651407 (+ 130us) server_negotiation.cc:300] Negotiation successful
1023 20:11:56.651437 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:11:56.651531 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:56.649299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:36981 (local address 127.9.117.193:48787)
1023 20:11:56.649461 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:56.649473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:11:56.649557 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:11:56.649687 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:11:56.649688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:11:56.649689 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:11:56.649876 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:56.649877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.650533 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:11:56.650534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:11:56.650969 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:11:56.650972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:11:56.651052 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:56.651058 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:11:56.651349 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:11:56.651352 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:11:56.651402 (+ 50us) client_negotiation.cc:240] Negotiation successful
1023 20:11:56.651456 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20251023 20:11:56.658150 11447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35790:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:56.658258 11447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35790:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:56.658993 11447 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: Sending DeleteTablet for 3 replicas of tablet e70c3bdd8f0d43308357dcba66cd2707
I20251023 20:11:56.659256 11586 tablet_service.cc:1552] Processing DeleteTablet for tablet e70c3bdd8f0d43308357dcba66cd2707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:48942
I20251023 20:11:56.659329 11871 tablet_service.cc:1552] Processing DeleteTablet for tablet e70c3bdd8f0d43308357dcba66cd2707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:47230
I20251023 20:11:56.661455 12032 tablet_replica.cc:333] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: stopping tablet replica
I20251023 20:11:56.661455 11729 tablet_service.cc:1552] Processing DeleteTablet for tablet e70c3bdd8f0d43308357dcba66cd2707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:58800
I20251023 20:11:56.661542 12032 raft_consensus.cc:2243] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:56.661594 12032 raft_consensus.cc:2272] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:56.661583 12033 tablet_replica.cc:333] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: stopping tablet replica
I20251023 20:11:56.661652 12033 raft_consensus.cc:2243] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:56.661674 12034 tablet_replica.cc:333] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: stopping tablet replica
I20251023 20:11:56.661692 12033 raft_consensus.cc:2272] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:56.661732 12034 raft_consensus.cc:2243] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:11:56.661847 12034 raft_consensus.cc:2272] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:56.662375 12033 ts_tablet_manager.cc:1916] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:56.662552 12032 ts_tablet_manager.cc:1916] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:56.662595 12034 ts_tablet_manager.cc:1916] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:56.664265 12033 ts_tablet_manager.cc:1929] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:56.664328 12033 log.cc:1199] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/e70c3bdd8f0d43308357dcba66cd2707
I20251023 20:11:56.664321 12032 ts_tablet_manager.cc:1929] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:56.664242 11447 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:11:56.664372 12032 log.cc:1199] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/e70c3bdd8f0d43308357dcba66cd2707
I20251023 20:11:56.664377 12034 ts_tablet_manager.cc:1929] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:11:56.664431 12034 log.cc:1199] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/e70c3bdd8f0d43308357dcba66cd2707
W20251023 20:11:56.664430 11447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:11:56.664597 12032 ts_tablet_manager.cc:1950] T e70c3bdd8f0d43308357dcba66cd2707 P 3e658244616547cfb168bd8d3875a8ff: Deleting consensus metadata
I20251023 20:11:56.664590 12033 ts_tablet_manager.cc:1950] T e70c3bdd8f0d43308357dcba66cd2707 P 4bc6a61bd8af495e9153879b96da40e3: Deleting consensus metadata
I20251023 20:11:56.664644 12034 ts_tablet_manager.cc:1950] T e70c3bdd8f0d43308357dcba66cd2707 P b2d9714c3e0346f080fd85e11e2a2602: Deleting consensus metadata
I20251023 20:11:56.665037 11434 catalog_manager.cc:4985] TS b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981): tablet e70c3bdd8f0d43308357dcba66cd2707 (table test-table [id=49ed46dd80064adeab4d0ed079b7a944]) successfully deleted
I20251023 20:11:56.665067 11433 catalog_manager.cc:4985] TS 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723): tablet e70c3bdd8f0d43308357dcba66cd2707 (table test-table [id=49ed46dd80064adeab4d0ed079b7a944]) successfully deleted
I20251023 20:11:56.665248 11432 catalog_manager.cc:4985] TS 3e658244616547cfb168bd8d3875a8ff (127.9.117.193:34245): tablet e70c3bdd8f0d43308357dcba66cd2707 (table test-table [id=49ed46dd80064adeab4d0ed079b7a944]) successfully deleted
I20251023 20:11:56.666736 11586 tablet_service.cc:1505] Processing CreateTablet for tablet 14885b49c5164aa2a9b9091185ef1cc4 (DEFAULT_TABLE table=test-table [id=f04ec0c9fa044c3987db5f4f69ce47ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:56.666775 11729 tablet_service.cc:1505] Processing CreateTablet for tablet 14885b49c5164aa2a9b9091185ef1cc4 (DEFAULT_TABLE table=test-table [id=f04ec0c9fa044c3987db5f4f69ce47ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:56.666795 11871 tablet_service.cc:1505] Processing CreateTablet for tablet 14885b49c5164aa2a9b9091185ef1cc4 (DEFAULT_TABLE table=test-table [id=f04ec0c9fa044c3987db5f4f69ce47ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:11:56.666865 11586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14885b49c5164aa2a9b9091185ef1cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:56.666896 11871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14885b49c5164aa2a9b9091185ef1cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:56.666894 11729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14885b49c5164aa2a9b9091185ef1cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:11:56.668128 12044 tablet_bootstrap.cc:492] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: Bootstrap starting.
I20251023 20:11:56.668128 12045 tablet_bootstrap.cc:492] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: Bootstrap starting.
I20251023 20:11:56.668242 12043 tablet_bootstrap.cc:492] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff: Bootstrap starting.
I20251023 20:11:56.668579 12043 tablet_bootstrap.cc:654] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:56.668884 12045 tablet_bootstrap.cc:654] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:56.668720 12044 tablet_bootstrap.cc:654] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: Neither blocks nor log segments found. Creating new log.
I20251023 20:11:56.669643 12043 tablet_bootstrap.cc:492] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff: No bootstrap required, opened a new log
I20251023 20:11:56.669688 12045 tablet_bootstrap.cc:492] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: No bootstrap required, opened a new log
I20251023 20:11:56.669696 12043 ts_tablet_manager.cc:1403] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:11:56.669728 12045 ts_tablet_manager.cc:1403] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:11:56.669847 12043 raft_consensus.cc:359] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.669917 12043 raft_consensus.cc:385] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:56.669894 12045 raft_consensus.cc:359] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.669937 12043 raft_consensus.cc:740] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Initialized, Role: FOLLOWER
I20251023 20:11:56.669955 12045 raft_consensus.cc:385] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:56.669978 12045 raft_consensus.cc:740] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc6a61bd8af495e9153879b96da40e3, State: Initialized, Role: FOLLOWER
I20251023 20:11:56.669987 12043 consensus_queue.cc:260] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.670022 12045 consensus_queue.cc:260] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.670087 12043 ts_tablet_manager.cc:1434] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:56.670117 12045 ts_tablet_manager.cc:1434] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:56.670691 12044 tablet_bootstrap.cc:492] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: No bootstrap required, opened a new log
I20251023 20:11:56.670746 12044 ts_tablet_manager.cc:1403] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:11:56.670931 12044 raft_consensus.cc:359] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.671007 12044 raft_consensus.cc:385] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:11:56.671031 12044 raft_consensus.cc:740] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d9714c3e0346f080fd85e11e2a2602, State: Initialized, Role: FOLLOWER
I20251023 20:11:56.671074 12044 consensus_queue.cc:260] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.671187 12044 ts_tablet_manager.cc:1434] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:11:56.961760 11971 raft_consensus.cc:493] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:11:56.961901 11971 raft_consensus.cc:515] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.962108 11971 leader_election.cc:290] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723), b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981)
I20251023 20:11:56.962343 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14885b49c5164aa2a9b9091185ef1cc4" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3" is_pre_election: true
I20251023 20:11:56.962423 11749 raft_consensus.cc:2468] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 0.
I20251023 20:11:56.962365 11891 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14885b49c5164aa2a9b9091185ef1cc4" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d9714c3e0346f080fd85e11e2a2602" is_pre_election: true
I20251023 20:11:56.962527 11891 raft_consensus.cc:2468] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 0.
I20251023 20:11:56.962605 11539 leader_election.cc:304] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e658244616547cfb168bd8d3875a8ff, 4bc6a61bd8af495e9153879b96da40e3; no voters:
I20251023 20:11:56.962699 11971 raft_consensus.cc:2804] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:11:56.962743 11971 raft_consensus.cc:493] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:11:56.962769 11971 raft_consensus.cc:3060] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:56.963626 11971 raft_consensus.cc:515] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.963763 11971 leader_election.cc:290] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 election: Requested vote from peers 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723), b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981)
I20251023 20:11:56.963963 11749 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14885b49c5164aa2a9b9091185ef1cc4" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc6a61bd8af495e9153879b96da40e3"
I20251023 20:11:56.963980 11891 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14885b49c5164aa2a9b9091185ef1cc4" candidate_uuid: "3e658244616547cfb168bd8d3875a8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d9714c3e0346f080fd85e11e2a2602"
I20251023 20:11:56.964035 11749 raft_consensus.cc:3060] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:56.964047 11891 raft_consensus.cc:3060] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:11:56.964663 11749 raft_consensus.cc:2468] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 1.
I20251023 20:11:56.964773 11891 raft_consensus.cc:2468] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e658244616547cfb168bd8d3875a8ff in term 1.
I20251023 20:11:56.964874 11539 leader_election.cc:304] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e658244616547cfb168bd8d3875a8ff, 4bc6a61bd8af495e9153879b96da40e3; no voters:
I20251023 20:11:56.964967 11971 raft_consensus.cc:2804] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:11:56.965018 11971 raft_consensus.cc:697] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [term 1 LEADER]: Becoming Leader. State: Replica: 3e658244616547cfb168bd8d3875a8ff, State: Running, Role: LEADER
I20251023 20:11:56.965071 11971 consensus_queue.cc:237] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } }
I20251023 20:11:56.965716 11448 catalog_manager.cc:5649] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e658244616547cfb168bd8d3875a8ff (127.9.117.193). New cstate: current_term: 1 leader_uuid: "3e658244616547cfb168bd8d3875a8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e658244616547cfb168bd8d3875a8ff" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 } health_report { overall_health: UNKNOWN } } }
I20251023 20:11:57.028264 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:57.026550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.9.117.193:34245)
1023 20:11:57.026589 (+ 39us) server_negotiation.cc:207] Beginning negotiation
1023 20:11:57.026591 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:11:57.026598 (+ 7us) server_negotiation.cc:408] Connection header received
1023 20:11:57.026622 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:11:57.026623 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:11:57.026631 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:11:57.026678 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:11:57.026912 (+ 234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:57.027373 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:11:57.027904 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:11:57.028020 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:11:57.028035 (+ 15us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:11:57.028058 (+ 23us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:11:57.028100 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1023 20:11:57.028156 (+ 56us) server_negotiation.cc:300] Negotiation successful
1023 20:11:57.028173 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251023 20:11:57.030663 11891 raft_consensus.cc:1275] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Refusing update from remote peer 3e658244616547cfb168bd8d3875a8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:57.030687 11749 raft_consensus.cc:1275] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Refusing update from remote peer 3e658244616547cfb168bd8d3875a8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:11:57.030906 11971 consensus_queue.cc:1048] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc6a61bd8af495e9153879b96da40e3" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:11:57.031009 12020 consensus_queue.cc:1048] T 14885b49c5164aa2a9b9091185ef1cc4 P 3e658244616547cfb168bd8d3875a8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d9714c3e0346f080fd85e11e2a2602" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:11:57.035740 12013 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251023 20:11:58.559226 11447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35790:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:58.559352 11447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35790:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:11:58.560525 11447 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b7f9ce12eadb432e97dc462e7add49c7: Sending DeleteTablet for 3 replicas of tablet 14885b49c5164aa2a9b9091185ef1cc4
I20251023 20:11:58.560876 11586 tablet_service.cc:1552] Processing DeleteTablet for tablet 14885b49c5164aa2a9b9091185ef1cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:58 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:48942
I20251023 20:11:58.560904 11729 tablet_service.cc:1552] Processing DeleteTablet for tablet 14885b49c5164aa2a9b9091185ef1cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:58 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:58800
I20251023 20:11:58.561025 11871 tablet_service.cc:1552] Processing DeleteTablet for tablet 14885b49c5164aa2a9b9091185ef1cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:11:58 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:47230
I20251023 20:11:58.561147 12095 tablet_replica.cc:333] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: stopping tablet replica
I20251023 20:11:58.561220 12095 raft_consensus.cc:2243] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:58.561275 12095 raft_consensus.cc:2272] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:58.561272 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 11522
I20251023 20:11:58.562434 12095 ts_tablet_manager.cc:1916] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:11:58.564674 12095 ts_tablet_manager.cc:1929] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251023 20:11:58.564779 12095 log.cc:1199] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/14885b49c5164aa2a9b9091185ef1cc4
I20251023 20:11:58.565011 12095 ts_tablet_manager.cc:1950] T 14885b49c5164aa2a9b9091185ef1cc4 P 4bc6a61bd8af495e9153879b96da40e3: Deleting consensus metadata
I20251023 20:11:58.565546 11433 catalog_manager.cc:4985] TS 4bc6a61bd8af495e9153879b96da40e3 (127.9.117.194:35723): tablet 14885b49c5164aa2a9b9091185ef1cc4 (table test-table [id=f04ec0c9fa044c3987db5f4f69ce47ad]) successfully deleted
I20251023 20:11:58.568893 12097 tablet_replica.cc:333] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: stopping tablet replica
I20251023 20:11:58.568985 12097 raft_consensus.cc:2243] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:11:58.569056 12097 raft_consensus.cc:2272] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:11:58.569491 12097 ts_tablet_manager.cc:1916] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:11:58.569762 11432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:34245 (error 108)
I20251023 20:11:58.570150 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 11664
I20251023 20:11:58.572880 12097 ts_tablet_manager.cc:1929] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251023 20:11:58.572971 12097 log.cc:1199] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/14885b49c5164aa2a9b9091185ef1cc4
I20251023 20:11:58.573040 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:11:58.570222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34245 (local address 127.0.0.1:46664)
1023 20:11:58.572901 (+ 2679us) negotiation.cc:107] Waiting for socket to connect
1023 20:11:58.572977 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:34245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2640,"thread_start_us":67,"threads_started":1}
W20251023 20:11:58.573225 11432 catalog_manager.cc:4712] TS 3e658244616547cfb168bd8d3875a8ff (127.9.117.193:34245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14885b49c5164aa2a9b9091185ef1cc4: Network error: Client connection negotiation failed: client connection to 127.9.117.193:34245: connect: Connection refused (error 111)
I20251023 20:11:58.573287 12097 ts_tablet_manager.cc:1950] T 14885b49c5164aa2a9b9091185ef1cc4 P b2d9714c3e0346f080fd85e11e2a2602: Deleting consensus metadata
I20251023 20:11:58.573706 11434 catalog_manager.cc:4985] TS b2d9714c3e0346f080fd85e11e2a2602 (127.9.117.195:36981): tablet 14885b49c5164aa2a9b9091185ef1cc4 (table test-table [id=f04ec0c9fa044c3987db5f4f69ce47ad]) successfully deleted
I20251023 20:11:58.575580 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 11806
I20251023 20:11:58.582396 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 11416
2025-10-23T20:11:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7678 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:11:58 dist-test-slave-x62p krb5kdc[12102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:11:58 dist-test-slave-x62p krb5kdc[12102](info): set up 2 sockets
Oct 23 20:11:58 dist-test-slave-x62p krb5kdc[12102](info): commencing operation
krb5kdc: starting...
W20251023 20:12:00.621539 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:00 dist-test-slave-x62p krb5kdc[12102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:00.785296 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:46035
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:46035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:00.872756 12118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:00.872963 12118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:00.872990 12118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:00.874493 12118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:00.874552 12118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:00.874567 12118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:00.874581 12118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:00.874593 12118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:00.876300 12118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:46035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:46035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:00.876549 12118 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:00.876757 12118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:00.880172 12123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:00.880189 12124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:00.880384 12126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:00.880474 12118 server_base.cc:1047] running on GCE node
I20251023 20:12:00.880781 12118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:00.881078 12118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:00.882236 12118 hybrid_clock.cc:648] HybridClock initialized: now 1761250320882215 us; error 33 us; skew 500 ppm
Oct 23 20:12:00 dist-test-slave-x62p krb5kdc[12102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250320, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:00.884620 12118 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:00.885203 12118 webserver.cc:492] Webserver started at http://127.9.117.254:45745/ using document root <none> and password file <none>
I20251023 20:12:00.885371 12118 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:00.885402 12118 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:00.885491 12118 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:00.886495 12118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "b3073b938a9547b2aca71c9373c103d2"
format_stamp: "Formatted at 2025-10-23 20:12:00 on dist-test-slave-x62p"
server_key: "eaca8541267e93993b5fd16af06ccb77"
server_key_iv: "f0b9647e4c413fddf0f94dba10e1d8f1"
server_key_version: "encryptionkey@0"
I20251023 20:12:00.886786 12118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "b3073b938a9547b2aca71c9373c103d2"
format_stamp: "Formatted at 2025-10-23 20:12:00 on dist-test-slave-x62p"
server_key: "eaca8541267e93993b5fd16af06ccb77"
server_key_iv: "f0b9647e4c413fddf0f94dba10e1d8f1"
server_key_version: "encryptionkey@0"
I20251023 20:12:00.888046 12118 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:00.888864 12133 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:00.889030 12118 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:00.889103 12118 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "b3073b938a9547b2aca71c9373c103d2"
format_stamp: "Formatted at 2025-10-23 20:12:00 on dist-test-slave-x62p"
server_key: "eaca8541267e93993b5fd16af06ccb77"
server_key_iv: "f0b9647e4c413fddf0f94dba10e1d8f1"
server_key_version: "encryptionkey@0"
I20251023 20:12:00.889173 12118 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:00.900353 12118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:00.902657 12118 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:00.902817 12118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:00.907394 12118 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:46035
I20251023 20:12:00.907469 12185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:46035 every 8 connection(s)
I20251023 20:12:00.908139 12118 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:00.908846 12186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:00.910356 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 12118
I20251023 20:12:00.910473 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:00.910650 9687 external_mini_cluster.cc:1468] Setting key c0e0af6b0c54b9b31175fb40da46e15d
I20251023 20:12:00.912158 12186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2: Bootstrap starting.
I20251023 20:12:00.912844 12186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:00.913121 12186 log.cc:826] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:00.913825 12186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2: No bootstrap required, opened a new log
I20251023 20:12:00.914953 12186 raft_consensus.cc:359] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3073b938a9547b2aca71c9373c103d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46035 } }
I20251023 20:12:00.915088 12186 raft_consensus.cc:385] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:00.915124 12186 raft_consensus.cc:740] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3073b938a9547b2aca71c9373c103d2, State: Initialized, Role: FOLLOWER
I20251023 20:12:00.915274 12186 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3073b938a9547b2aca71c9373c103d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46035 } }
I20251023 20:12:00.915338 12186 raft_consensus.cc:399] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:00.915372 12186 raft_consensus.cc:493] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:00.915422 12186 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:00.916044 12186 raft_consensus.cc:515] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3073b938a9547b2aca71c9373c103d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46035 } }
I20251023 20:12:00.916165 12186 leader_election.cc:304] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3073b938a9547b2aca71c9373c103d2; no voters:
I20251023 20:12:00.916316 12186 leader_election.cc:290] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:00.916360 12191 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:00.916486 12191 raft_consensus.cc:697] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [term 1 LEADER]: Becoming Leader. State: Replica: b3073b938a9547b2aca71c9373c103d2, State: Running, Role: LEADER
Oct 23 20:12:00 dist-test-slave-x62p krb5kdc[12102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:00.916584 12191 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3073b938a9547b2aca71c9373c103d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46035 } }
I20251023 20:12:00.916608 12186 sys_catalog.cc:565] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:00.917574 12192 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3073b938a9547b2aca71c9373c103d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3073b938a9547b2aca71c9373c103d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46035 } } }
I20251023 20:12:00.917610 12193 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3073b938a9547b2aca71c9373c103d2. Latest consensus state: current_term: 1 leader_uuid: "b3073b938a9547b2aca71c9373c103d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3073b938a9547b2aca71c9373c103d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46035 } } }
I20251023 20:12:00.917685 12192 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:00.917686 12193 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:00.917932 12200 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:00.918773 12200 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:00.920430 12200 catalog_manager.cc:1357] Generated new cluster ID: c3dc1c503ae845ea9f7eac8e8ac1378d
I20251023 20:12:00.920485 12200 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:00.921394 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:00.911265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.9.117.254:46035)
1023 20:12:00.911457 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:00.911469 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:00.911490 (+ 21us) server_negotiation.cc:408] Connection header received
1023 20:12:00.911850 (+ 360us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:00.911856 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:00.911904 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:00.912039 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:00.913115 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:00.913910 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:00.914474 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:00.915110 (+ 636us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:00.917094 (+ 1984us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:00.917102 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:00.917103 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:00.917122 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:00.919517 (+ 2395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:00.919921 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:00.919925 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:00.919925 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:00.919993 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:00.920181 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:00.920182 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:00.920183 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:00.920451 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:00.920620 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:00.921101 (+ 481us) server_negotiation.cc:300] Negotiation successful
1023 20:12:00.921220 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20251023 20:12:00.940652 12200 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:00.941279 12200 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:00.946167 12200 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b3073b938a9547b2aca71c9373c103d2: Generated new TSK 0
I20251023 20:12:00.946377 12200 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:01.006019 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46035
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:01.091054 12214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:01.091229 12214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:01.091255 12214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:01.092645 12214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:01.092687 12214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:01.092715 12214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:01.094336 12214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:01.094549 12214 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:01.094710 12214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:01.097409 12219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:01.097412 12222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:01.097628 12214 server_base.cc:1047] running on GCE node
W20251023 20:12:01.097702 12220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:01.097882 12214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:01.098057 12214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:01.099211 12214 hybrid_clock.cc:648] HybridClock initialized: now 1761250321099195 us; error 30 us; skew 500 ppm
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:01.101852 12214 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:12:01.102398 12214 webserver.cc:492] Webserver started at http://127.9.117.193:42555/ using document root <none> and password file <none>
I20251023 20:12:01.102603 12214 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:01.102644 12214 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:01.102761 12214 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:01.103775 12214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "5e40c9e5883c41d3bf3eb9020fdb85c1"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "034c4c98d1f968a772cac042f1f6aee4"
server_key_iv: "0e5dd9f76b02a8d52b04b38efa712d8a"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.104085 12214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "5e40c9e5883c41d3bf3eb9020fdb85c1"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "034c4c98d1f968a772cac042f1f6aee4"
server_key_iv: "0e5dd9f76b02a8d52b04b38efa712d8a"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.105361 12214 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:01.106130 12229 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.106334 12214 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:12:01.106408 12214 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "5e40c9e5883c41d3bf3eb9020fdb85c1"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "034c4c98d1f968a772cac042f1f6aee4"
server_key_iv: "0e5dd9f76b02a8d52b04b38efa712d8a"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.106467 12214 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:01.113310 12214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:01.115253 12214 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:01.115378 12214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:01.115592 12214 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:01.115897 12214 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:01.115929 12214 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.115962 12214 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:01.115989 12214 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.122315 12214 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:34185
I20251023 20:12:01.122354 12342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:34185 every 8 connection(s)
I20251023 20:12:01.122741 12214 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:01.130802 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 12214
I20251023 20:12:01.130911 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:01.131068 9687 external_mini_cluster.cc:1468] Setting key 296666b2fbd3428d58e0ea68dbdc84ce
I20251023 20:12:01.132046 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:01.123555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:59923 (local address 127.9.117.254:46035)
1023 20:12:01.123611 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:01.123613 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:01.123778 (+ 165us) server_negotiation.cc:408] Connection header received
1023 20:12:01.124253 (+ 475us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:01.124254 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:01.124264 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:01.124318 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:01.125135 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.125702 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:01.126818 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.126966 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:01.128876 (+ 1910us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:01.128881 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:01.128881 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:01.128888 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:01.130424 (+ 1536us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:01.130955 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:01.130957 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:01.130958 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:01.130990 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:01.131246 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:01.131247 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:01.131247 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:01.131371 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:01.131443 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:01.131816 (+ 373us) server_negotiation.cc:300] Negotiation successful
1023 20:12:01.131923 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:12:01.132120 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:01.123459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46035 (local address 127.9.117.193:59923)
1023 20:12:01.123674 (+ 215us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:01.123699 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:01.124183 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:01.124365 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:01.124369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:01.124689 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:01.125023 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:01.125028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.125804 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:01.125805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:01.126721 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:01.126723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.126845 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:01.127367 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:01.127387 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:01.128775 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:01.130516 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:01.130522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:01.130525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:01.130854 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:01.131062 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:01.131063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:01.131064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:01.131159 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:01.131464 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:01.131471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:01.131643 (+ 172us) client_negotiation.cc:769] Sending connection context
1023 20:12:01.131805 (+ 162us) client_negotiation.cc:240] Negotiation successful
1023 20:12:01.131962 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20251023 20:12:01.132642 12343 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46035
I20251023 20:12:01.132812 12343 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:01.133066 12343 heartbeater.cc:507] Master 127.9.117.254:46035 requested a full tablet report, sending...
I20251023 20:12:01.133697 12150 ts_manager.cc:194] Registered new tserver with Master: 5e40c9e5883c41d3bf3eb9020fdb85c1 (127.9.117.193:34185)
I20251023 20:12:01.134456 12150 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:59923
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:01.184124 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46035
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:01.265826 12350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:01.266014 12350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:01.266052 12350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:01.267485 12350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:01.267544 12350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:01.267588 12350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:01.269157 12350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:01.269411 12350 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:01.269603 12350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:01.272200 12356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:01.272326 12355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:01.272336 12358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:01.272457 12350 server_base.cc:1047] running on GCE node
I20251023 20:12:01.272660 12350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:01.272912 12350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:01.274041 12350 hybrid_clock.cc:648] HybridClock initialized: now 1761250321274028 us; error 31 us; skew 500 ppm
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:01.276350 12350 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:12:01.276959 12350 webserver.cc:492] Webserver started at http://127.9.117.194:40441/ using document root <none> and password file <none>
I20251023 20:12:01.277145 12350 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:01.277184 12350 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:01.277298 12350 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:01.278350 12350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "4a6bfdc731d047638b3da9adf4cade02"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "ab99884d4db520f0ddb3565ebbb1e8b0"
server_key_iv: "5a4de868dff9052d691e68363633344b"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.278736 12350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "4a6bfdc731d047638b3da9adf4cade02"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "ab99884d4db520f0ddb3565ebbb1e8b0"
server_key_iv: "5a4de868dff9052d691e68363633344b"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.279970 12350 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:12:01.280652 12365 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.280886 12350 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:01.280964 12350 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "4a6bfdc731d047638b3da9adf4cade02"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "ab99884d4db520f0ddb3565ebbb1e8b0"
server_key_iv: "5a4de868dff9052d691e68363633344b"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.281036 12350 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:01.319474 12350 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:01.321764 12350 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:01.321929 12350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:01.322181 12350 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:01.322542 12350 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:01.322577 12350 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.322614 12350 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:01.322644 12350 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.328788 12350 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:42161
I20251023 20:12:01.328846 12478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:42161 every 8 connection(s)
I20251023 20:12:01.329154 12350 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:12:01.329442 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 12350
I20251023 20:12:01.329540 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:01.329710 9687 external_mini_cluster.cc:1468] Setting key 81b3a267679f0adaf7997c74919bc29a
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
I20251023 20:12:01.338884 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:01.329881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:54943 (local address 127.9.117.254:46035)
1023 20:12:01.329987 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:01.329990 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:01.330145 (+ 155us) server_negotiation.cc:408] Connection header received
1023 20:12:01.330662 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:01.330664 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:01.330676 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:01.330732 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:01.331509 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.332124 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:01.333611 (+ 1487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.333770 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:01.336190 (+ 2420us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:01.336195 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:01.336195 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:01.336204 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:01.337624 (+ 1420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:01.338102 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:01.338103 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:01.338103 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:01.338127 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:01.338278 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:01.338279 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:01.338279 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:01.338383 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:01.338432 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:01.338691 (+ 259us) server_negotiation.cc:300] Negotiation successful
1023 20:12:01.338801 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251023 20:12:01.338958 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:01.329795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46035 (local address 127.9.117.194:54943)
1023 20:12:01.330029 (+ 234us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:01.330060 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:01.330582 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:01.330807 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:01.330810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:01.331123 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:01.331428 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:01.331431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.332216 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:01.332217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:01.333469 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:01.333473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.333683 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:01.334268 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:01.334298 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:01.336096 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:01.337709 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:01.337717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:01.337721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:01.338031 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:01.338181 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:01.338181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:01.338182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:01.338238 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:01.338425 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:01.338428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:01.338580 (+ 152us) client_negotiation.cc:769] Sending connection context
1023 20:12:01.338674 (+ 94us) client_negotiation.cc:240] Negotiation successful
1023 20:12:01.338799 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20251023 20:12:01.339460 12479 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46035
I20251023 20:12:01.339557 12479 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:01.339788 12479 heartbeater.cc:507] Master 127.9.117.254:46035 requested a full tablet report, sending...
I20251023 20:12:01.340183 12150 ts_manager.cc:194] Registered new tserver with Master: 4a6bfdc731d047638b3da9adf4cade02 (127.9.117.194:42161)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
I20251023 20:12:01.340744 12150 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:54943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:01.379325 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46035
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:01.468591 12486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:01.468861 12486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:01.468905 12486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:01.471517 12486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:01.471602 12486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:01.471660 12486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:01.474607 12486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:01.474920 12486 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:01.475134 12486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:01.478129 12491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:01.478166 12494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:01.478145 12492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:01.478590 12486 server_base.cc:1047] running on GCE node
I20251023 20:12:01.478801 12486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:01.479050 12486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:01.482318 12486 hybrid_clock.cc:648] HybridClock initialized: now 1761250321482286 us; error 41 us; skew 500 ppm
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:01.485790 12486 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:12:01.486438 12486 webserver.cc:492] Webserver started at http://127.9.117.195:40941/ using document root <none> and password file <none>
I20251023 20:12:01.486635 12486 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:01.486676 12486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:01.486815 12486 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:01.487856 12486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "7538aa22c52445768ce677c48d9a7332"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "027c2c98b41ce3d6af9cc56832eaa63c"
server_key_iv: "8bd77f24374538f7d6e7d01b2f01a61e"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.488168 12486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "7538aa22c52445768ce677c48d9a7332"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "027c2c98b41ce3d6af9cc56832eaa63c"
server_key_iv: "8bd77f24374538f7d6e7d01b2f01a61e"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.489681 12486 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:01.490541 12501 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.490736 12486 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:01.490809 12486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "7538aa22c52445768ce677c48d9a7332"
format_stamp: "Formatted at 2025-10-23 20:12:01 on dist-test-slave-x62p"
server_key: "027c2c98b41ce3d6af9cc56832eaa63c"
server_key_iv: "8bd77f24374538f7d6e7d01b2f01a61e"
server_key_version: "encryptionkey@0"
I20251023 20:12:01.490876 12486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:01.510974 12486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:01.513262 12486 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:01.513403 12486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:01.513643 12486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:01.513969 12486 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:01.514003 12486 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.514036 12486 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:01.514063 12486 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:01.520112 12486 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:33221
I20251023 20:12:01.520171 12614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:33221 every 8 connection(s)
I20251023 20:12:01.520452 12486 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:12:01.525730 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 12486
I20251023 20:12:01.525813 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:01.525964 9687 external_mini_cluster.cc:1468] Setting key 285606b29e36c9fc85b6ef4218c08c16
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250321, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:01.529503 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:01.521149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:42443 (local address 127.9.117.254:46035)
1023 20:12:01.521204 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:01.521206 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:01.521341 (+ 135us) server_negotiation.cc:408] Connection header received
1023 20:12:01.521862 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:01.521863 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:01.521872 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:01.521934 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:01.522730 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.523229 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:01.524326 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.524454 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:01.526669 (+ 2215us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:01.526674 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:01.526675 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:01.526684 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:01.528136 (+ 1452us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:01.528536 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:01.528537 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:01.528537 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:01.528566 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:01.528803 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:01.528804 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:01.528804 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:01.528914 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:01.528987 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:01.529311 (+ 324us) server_negotiation.cc:300] Negotiation successful
1023 20:12:01.529407 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:12:01.529558 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:01.521100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46035 (local address 127.9.117.195:42443)
1023 20:12:01.521248 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:01.521278 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:01.521784 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:01.521991 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:01.521995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:01.522314 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:01.522613 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:01.522616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.523340 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:01.523341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:01.524233 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:01.524235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:01.524361 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:01.524972 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:01.524992 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:01.526568 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:01.528218 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:01.528224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:01.528226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:01.528443 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:01.528629 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:01.528629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:01.528629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:01.528685 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:01.529005 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:01.529011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:01.529152 (+ 141us) client_negotiation.cc:769] Sending connection context
1023 20:12:01.529268 (+ 116us) client_negotiation.cc:240] Negotiation successful
1023 20:12:01.529394 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20251023 20:12:01.530166 12615 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46035
I20251023 20:12:01.530285 12615 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:01.530560 12615 heartbeater.cc:507] Master 127.9.117.254:46035 requested a full tablet report, sending...
I20251023 20:12:01.530961 12150 ts_manager.cc:194] Registered new tserver with Master: 7538aa22c52445768ce677c48d9a7332 (127.9.117.195:33221)
I20251023 20:12:01.531342 12150 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:42443
I20251023 20:12:01.537890 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251023 20:12:01.548309 12189 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:01.547443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.9.117.254:46035)
1023 20:12:01.547521 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:01.547523 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:01.547592 (+ 69us) server_negotiation.cc:408] Connection header received
1023 20:12:01.547655 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:01.547656 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:01.547666 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:01.547718 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:01.548248 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51258: BlockingRecv error: recv got EOF from 127.0.0.1:51258 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:12:01.549424 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 12214
I20251023 20:12:01.554666 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 12350
I20251023 20:12:01.559891 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 12486
I20251023 20:12:01.564646 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 12118
2025-10-23T20:12:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2980 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12627](info): set up 2 sockets
Oct 23 20:12:01 dist-test-slave-x62p krb5kdc[12627](info): commencing operation
krb5kdc: starting...
W20251023 20:12:03.563609 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:03 dist-test-slave-x62p krb5kdc[12627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:03.707758 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40357
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:40357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:03.791415 12643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:03.791638 12643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:03.791663 12643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:03.793366 12643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:03.793426 12643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:03.793440 12643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:03.793454 12643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:03.793468 12643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:03.795217 12643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:40357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:03.795471 12643 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:03.795670 12643 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:03.798550 12649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:03.798569 12651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:03.798557 12643 server_base.cc:1047] running on GCE node
W20251023 20:12:03.798533 12648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:03.798952 12643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:03.799208 12643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:03.800354 12643 hybrid_clock.cc:648] HybridClock initialized: now 1761250323800323 us; error 44 us; skew 500 ppm
Oct 23 20:12:03 dist-test-slave-x62p krb5kdc[12627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250323, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:03.802908 12643 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:03.803475 12643 webserver.cc:492] Webserver started at http://127.9.117.254:34823/ using document root <none> and password file <none>
I20251023 20:12:03.803676 12643 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:03.803716 12643 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:03.803825 12643 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:03.804870 12643 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "6573ac24dedd4b028f8068e513d7673d"
format_stamp: "Formatted at 2025-10-23 20:12:03 on dist-test-slave-x62p"
server_key: "fdb4348d2d6e89809719d78edf1c9ab8"
server_key_iv: "4f28f9d1424657c955ce2432503adcf0"
server_key_version: "encryptionkey@0"
I20251023 20:12:03.805174 12643 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "6573ac24dedd4b028f8068e513d7673d"
format_stamp: "Formatted at 2025-10-23 20:12:03 on dist-test-slave-x62p"
server_key: "fdb4348d2d6e89809719d78edf1c9ab8"
server_key_iv: "4f28f9d1424657c955ce2432503adcf0"
server_key_version: "encryptionkey@0"
I20251023 20:12:03.806406 12643 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:03.807361 12658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:03.807657 12643 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:03.807749 12643 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "6573ac24dedd4b028f8068e513d7673d"
format_stamp: "Formatted at 2025-10-23 20:12:03 on dist-test-slave-x62p"
server_key: "fdb4348d2d6e89809719d78edf1c9ab8"
server_key_iv: "4f28f9d1424657c955ce2432503adcf0"
server_key_version: "encryptionkey@0"
I20251023 20:12:03.807843 12643 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:03.839241 12643 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:03.845731 12643 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:03.845914 12643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:03.850164 12643 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:40357
I20251023 20:12:03.850195 12710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:40357 every 8 connection(s)
I20251023 20:12:03.850564 12643 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:03.851227 12711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:03.853068 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 12643
I20251023 20:12:03.853188 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:03.853363 9687 external_mini_cluster.cc:1468] Setting key d79e1ea70744a3aabd33fda4f536b092
I20251023 20:12:03.854558 12711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d: Bootstrap starting.
I20251023 20:12:03.855370 12711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:03.855711 12711 log.cc:826] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:03.856442 12711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d: No bootstrap required, opened a new log
I20251023 20:12:03.858057 12711 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6573ac24dedd4b028f8068e513d7673d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40357 } }
I20251023 20:12:03.858237 12711 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:03.858278 12711 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6573ac24dedd4b028f8068e513d7673d, State: Initialized, Role: FOLLOWER
I20251023 20:12:03.858397 12711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6573ac24dedd4b028f8068e513d7673d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40357 } }
I20251023 20:12:03.858479 12711 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:03.858520 12711 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:03.858577 12711 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:03.859460 12711 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6573ac24dedd4b028f8068e513d7673d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40357 } }
I20251023 20:12:03.859571 12711 leader_election.cc:304] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6573ac24dedd4b028f8068e513d7673d; no voters:
I20251023 20:12:03.859745 12711 leader_election.cc:290] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:03.859826 12716 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 1 FOLLOWER]: Leader election won for term 1
Oct 23 20:12:03 dist-test-slave-x62p krb5kdc[12627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:03.859997 12716 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [term 1 LEADER]: Becoming Leader. State: Replica: 6573ac24dedd4b028f8068e513d7673d, State: Running, Role: LEADER
I20251023 20:12:03.860143 12716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6573ac24dedd4b028f8068e513d7673d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40357 } }
I20251023 20:12:03.860019 12711 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:03.861358 12717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6573ac24dedd4b028f8068e513d7673d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6573ac24dedd4b028f8068e513d7673d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40357 } } }
I20251023 20:12:03.861470 12717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:03.861773 12725 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:03.862408 12718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6573ac24dedd4b028f8068e513d7673d. Latest consensus state: current_term: 1 leader_uuid: "6573ac24dedd4b028f8068e513d7673d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6573ac24dedd4b028f8068e513d7673d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40357 } } }
I20251023 20:12:03.863238 12718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:03.863371 12725 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:03.864722 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:03.854006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.9.117.254:40357)
1023 20:12:03.854199 (+ 193us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:03.854209 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:03.854231 (+ 22us) server_negotiation.cc:408] Connection header received
1023 20:12:03.854567 (+ 336us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:03.854573 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:03.854623 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:03.854756 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:03.855896 (+ 1140us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:03.856717 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:03.857612 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:03.857862 (+ 250us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:03.860482 (+ 2620us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:03.860492 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:03.860492 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:03.860522 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:03.862847 (+ 2325us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:03.863300 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:03.863303 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:03.863304 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:03.863378 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:03.863615 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:03.863617 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:03.863617 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:03.863925 (+ 308us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:03.864111 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:03.864342 (+ 231us) server_negotiation.cc:300] Negotiation successful
1023 20:12:03.864550 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20251023 20:12:03.865044 12725 catalog_manager.cc:1357] Generated new cluster ID: 41127810fe5b4afa8999ba623e687822
I20251023 20:12:03.865121 12725 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:03.871423 12725 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:03.872357 12725 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:03.877871 12725 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6573ac24dedd4b028f8068e513d7673d: Generated new TSK 0
I20251023 20:12:03.878085 12725 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:03.936852 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40357
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:04.035027 12739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:04.035233 12739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:04.035274 12739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:04.036767 12739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:04.036823 12739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:04.036871 12739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:04.038537 12739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:04.038806 12739 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:04.038998 12739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:12:04.041816 12739 server_base.cc:1047] running on GCE node
W20251023 20:12:04.041792 12744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:04.041797 12745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:04.042114 12747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:04.042277 12739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:04.042490 12739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:04.043629 12739 hybrid_clock.cc:648] HybridClock initialized: now 1761250324043616 us; error 29 us; skew 500 ppm
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[12627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:04.046109 12739 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:12:04.046638 12739 webserver.cc:492] Webserver started at http://127.9.117.193:42145/ using document root <none> and password file <none>
I20251023 20:12:04.046809 12739 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:04.046844 12739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:04.046934 12739 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:04.047927 12739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "8d5c391043df48eea83d9cc81c49eb28"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "489d8fa6b82b0d1ec4326a9ea7ac28ce"
server_key_iv: "268a3361d42f4c6fdfe127eeef1a7b14"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.048202 12739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "8d5c391043df48eea83d9cc81c49eb28"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "489d8fa6b82b0d1ec4326a9ea7ac28ce"
server_key_iv: "268a3361d42f4c6fdfe127eeef1a7b14"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.049477 12739 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:04.050141 12754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.050314 12739 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:12:04.050380 12739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "8d5c391043df48eea83d9cc81c49eb28"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "489d8fa6b82b0d1ec4326a9ea7ac28ce"
server_key_iv: "268a3361d42f4c6fdfe127eeef1a7b14"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.050434 12739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:04.068039 12739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:04.070001 12739 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:04.070122 12739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:04.070325 12739 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:04.070612 12739 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:04.070643 12739 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.070665 12739 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:04.070683 12739 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.076454 12739 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:43643
I20251023 20:12:04.076556 12867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:43643 every 8 connection(s)
I20251023 20:12:04.076840 12739 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:04.082432 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 12739
I20251023 20:12:04.082543 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[12627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:04.082717 9687 external_mini_cluster.cc:1468] Setting key 62b7a58c92012734ee1840b48d8602e4
I20251023 20:12:04.086508 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:04.077622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:51031 (local address 127.9.117.254:40357)
1023 20:12:04.077708 (+ 86us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:04.077710 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:04.077860 (+ 150us) server_negotiation.cc:408] Connection header received
1023 20:12:04.078392 (+ 532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:04.078393 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:04.078403 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:04.078447 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:04.079333 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.079846 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:04.080455 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.080619 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:04.083174 (+ 2555us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:04.083180 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:04.083181 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:04.083189 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:04.085201 (+ 2012us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:04.085611 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:04.085613 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:04.085613 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:04.085646 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:04.085816 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:04.085817 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:04.085818 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:04.085957 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:04.086032 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:04.086258 (+ 226us) server_negotiation.cc:300] Negotiation successful
1023 20:12:04.086393 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251023 20:12:04.086540 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:04.077516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40357 (local address 127.9.117.193:51031)
1023 20:12:04.077711 (+ 195us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:04.077773 (+ 62us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:04.078296 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:04.078510 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:04.078513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:04.078854 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:04.079155 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:04.079159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.079976 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:04.079977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:04.080359 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:04.080360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.080477 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:04.081515 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:04.081533 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:04.083099 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:04.085309 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:04.085315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:04.085317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:04.085530 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:04.085704 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:04.085705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:04.085705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:04.085754 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:04.086012 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:04.086015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:04.086135 (+ 120us) client_negotiation.cc:769] Sending connection context
1023 20:12:04.086222 (+ 87us) client_negotiation.cc:240] Negotiation successful
1023 20:12:04.086336 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20251023 20:12:04.087214 12868 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40357
I20251023 20:12:04.087342 12868 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:04.087625 12868 heartbeater.cc:507] Master 127.9.117.254:40357 requested a full tablet report, sending...
I20251023 20:12:04.088158 12675 ts_manager.cc:194] Registered new tserver with Master: 8d5c391043df48eea83d9cc81c49eb28 (127.9.117.193:43643)
I20251023 20:12:04.088884 12675 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:51031
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:04.133388 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40357
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:04.219106 12875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:04.219332 12875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:04.219383 12875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:04.220950 12875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:04.221007 12875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:04.221051 12875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:04.222792 12875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:04.223073 12875 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:04.223271 12875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:04.226053 12883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:04.226069 12880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:04.226200 12875 server_base.cc:1047] running on GCE node
W20251023 20:12:04.226073 12881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:04.226454 12875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:04.226676 12875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:04.227815 12875 hybrid_clock.cc:648] HybridClock initialized: now 1761250324227804 us; error 33 us; skew 500 ppm
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[12627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:04.230361 12875 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:12:04.230968 12875 webserver.cc:492] Webserver started at http://127.9.117.194:41143/ using document root <none> and password file <none>
I20251023 20:12:04.231171 12875 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:04.231210 12875 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:04.231329 12875 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:04.232324 12875 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "0237212082eb4d0d99da60c2191326cb"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "83ab2686fef281ffad40928eb539c441"
server_key_iv: "89ee75498057c95929f2ceece3678a83"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.232636 12875 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "0237212082eb4d0d99da60c2191326cb"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "83ab2686fef281ffad40928eb539c441"
server_key_iv: "89ee75498057c95929f2ceece3678a83"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.233944 12875 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:04.234606 12890 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.234790 12875 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:12:04.234865 12875 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "0237212082eb4d0d99da60c2191326cb"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "83ab2686fef281ffad40928eb539c441"
server_key_iv: "89ee75498057c95929f2ceece3678a83"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.234930 12875 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:04.264706 12875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:04.266484 12875 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:04.266621 12875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:04.266867 12875 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:04.267199 12875 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:04.267234 12875 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.267270 12875 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:04.267302 12875 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.272933 12875 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:46535
I20251023 20:12:04.273000 13003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:46535 every 8 connection(s)
I20251023 20:12:04.273278 12875 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:12:04.278441 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 12875
I20251023 20:12:04.278584 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:04.278785 9687 external_mini_cluster.cc:1468] Setting key a9810cacd4d8abd5876ab8a49f13ee6b
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[12627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:04.282346 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:04.273974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:50455 (local address 127.9.117.254:40357)
1023 20:12:04.274028 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:04.274030 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:04.274195 (+ 165us) server_negotiation.cc:408] Connection header received
1023 20:12:04.274672 (+ 477us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:04.274673 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:04.274688 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:04.274754 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:04.275569 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.276108 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:04.276696 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.276893 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:04.279449 (+ 2556us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:04.279452 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:04.279453 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:04.279461 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:04.280980 (+ 1519us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:04.281375 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:04.281376 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:04.281377 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:04.281412 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:04.281578 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:04.281579 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:04.281579 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:04.281707 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:04.281790 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:04.282083 (+ 293us) server_negotiation.cc:300] Negotiation successful
1023 20:12:04.282207 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:12:04.282357 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:04.273923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40357 (local address 127.9.117.194:50455)
1023 20:12:04.274081 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:04.274113 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:04.274594 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:04.274822 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:04.274825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:04.275152 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:04.275449 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:04.275451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.276201 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:04.276202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:04.276593 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:04.276594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.276710 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:04.277639 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:04.277656 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:04.279372 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:04.281074 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:04.281082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:04.281086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:04.281306 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:04.281468 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:04.281468 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:04.281468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:04.281523 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:04.281798 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:04.281804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:04.281944 (+ 140us) client_negotiation.cc:769] Sending connection context
1023 20:12:04.282052 (+ 108us) client_negotiation.cc:240] Negotiation successful
1023 20:12:04.282178 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20251023 20:12:04.282976 13004 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40357
I20251023 20:12:04.283088 13004 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:04.283298 13004 heartbeater.cc:507] Master 127.9.117.254:40357 requested a full tablet report, sending...
I20251023 20:12:04.283676 12675 ts_manager.cc:194] Registered new tserver with Master: 0237212082eb4d0d99da60c2191326cb (127.9.117.194:46535)
I20251023 20:12:04.284065 12675 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:50455
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:04.327601 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40357
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:04.410564 13011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:04.410790 13011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:04.410830 13011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:04.412411 13011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:04.412464 13011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:04.412508 13011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:04.414278 13011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:04.414551 13011 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:04.414736 13011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:04.417515 13016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:04.417569 13017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:04.417577 13019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:04.417629 13011 server_base.cc:1047] running on GCE node
I20251023 20:12:04.417878 13011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:04.418071 13011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:04.419217 13011 hybrid_clock.cc:648] HybridClock initialized: now 1761250324419200 us; error 28 us; skew 500 ppm
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[12627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:04.422010 13011 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:12:04.422572 13011 webserver.cc:492] Webserver started at http://127.9.117.195:38645/ using document root <none> and password file <none>
I20251023 20:12:04.422791 13011 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:04.422832 13011 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:04.422930 13011 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:04.423908 13011 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "99e76f1f86994545b60dcc354ee111b4"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "e94100fd25160aa17148391c9411f1f1"
server_key_iv: "c3b912f6069b20bbb882d5dc2cbafdf8"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.424189 13011 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "99e76f1f86994545b60dcc354ee111b4"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "e94100fd25160aa17148391c9411f1f1"
server_key_iv: "c3b912f6069b20bbb882d5dc2cbafdf8"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.425483 13011 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:04.426249 13026 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.426456 13011 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:04.426544 13011 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "99e76f1f86994545b60dcc354ee111b4"
format_stamp: "Formatted at 2025-10-23 20:12:04 on dist-test-slave-x62p"
server_key: "e94100fd25160aa17148391c9411f1f1"
server_key_iv: "c3b912f6069b20bbb882d5dc2cbafdf8"
server_key_version: "encryptionkey@0"
I20251023 20:12:04.426610 13011 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:04.434175 13011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:04.436072 13011 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:04.436205 13011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:04.436412 13011 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:04.436704 13011 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:04.436794 13011 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.436833 13011 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:04.436856 13011 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:04.443120 13011 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:40057
I20251023 20:12:04.443199 13139 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:40057 every 8 connection(s)
I20251023 20:12:04.443473 13011 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[12627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250324, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:04.452368 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 13011
I20251023 20:12:04.452486 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:04.452642 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:04.444353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:48571 (local address 127.9.117.254:40357)
1023 20:12:04.444427 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:04.444430 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:04.444579 (+ 149us) server_negotiation.cc:408] Connection header received
1023 20:12:04.445188 (+ 609us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:04.445189 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:04.445204 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:04.445262 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:04.446089 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.446592 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:04.447149 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.447320 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:04.449906 (+ 2586us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:04.449912 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:04.449912 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:04.449922 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:04.451351 (+ 1429us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:04.451774 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:04.451775 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:04.451775 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:04.451795 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:04.451950 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:04.451950 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:04.451954 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:04.452055 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:04.452100 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:04.452415 (+ 315us) server_negotiation.cc:300] Negotiation successful
1023 20:12:04.452550 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251023 20:12:04.452673 9687 external_mini_cluster.cc:1468] Setting key c36b2ad70f3c208b5b621336be3bdbdb
I20251023 20:12:04.452873 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:04.444292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40357 (local address 127.9.117.195:48571)
1023 20:12:04.444459 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:04.444486 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:04.445080 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:04.445329 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:04.445333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:04.445688 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:04.445986 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:04.445990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.446674 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:04.446675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:04.447061 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:04.447063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:04.447211 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:04.448202 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:04.448220 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:04.449792 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:04.451458 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:04.451466 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:04.451469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:04.451710 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:04.451844 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:04.451844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:04.451844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:04.451905 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:04.452098 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:04.452101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:04.452272 (+ 171us) client_negotiation.cc:769] Sending connection context
1023 20:12:04.452413 (+ 141us) client_negotiation.cc:240] Negotiation successful
1023 20:12:04.452593 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20251023 20:12:04.453341 13140 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40357
I20251023 20:12:04.453457 13140 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:04.453733 13140 heartbeater.cc:507] Master 127.9.117.254:40357 requested a full tablet report, sending...
I20251023 20:12:04.454191 12674 ts_manager.cc:194] Registered new tserver with Master: 99e76f1f86994545b60dcc354ee111b4 (127.9.117.195:40057)
I20251023 20:12:04.454653 12674 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:48571
I20251023 20:12:04.465034 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251023 20:12:04.576000 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:04.574058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.9.117.254:40357)
1023 20:12:04.574148 (+ 90us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:04.574150 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:04.574333 (+ 183us) server_negotiation.cc:408] Connection header received
1023 20:12:04.574868 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:04.574870 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:04.574883 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:04.574954 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:04.575943 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53062: BlockingRecv error: recv got EOF from 127.0.0.1:53062 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20251023 20:12:04.576900 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:04.576232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.9.117.254:40357)
1023 20:12:04.576288 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:04.576291 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:04.576375 (+ 84us) server_negotiation.cc:408] Connection header received
1023 20:12:04.576426 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:04.576427 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:04.576444 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:04.576518 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:04.576838 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53076: BlockingRecv error: recv got EOF from 127.0.0.1:53076 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20251023 20:12:04.577481 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:04.577027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.9.117.254:40357)
1023 20:12:04.577063 (+ 36us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:04.577065 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:04.577074 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:04.577096 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:04.577097 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:04.577107 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:04.577158 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:04.577422 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53082: BlockingRecv error: recv got EOF from 127.0.0.1:53082 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20251023 20:12:04.579141 12714 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:04.578655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.9.117.254:40357)
1023 20:12:04.578709 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:04.578711 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:04.578720 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:04.578767 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:04.578768 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:04.578790 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:04.578832 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:04.579107 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53086: BlockingRecv error: recv got EOF from 127.0.0.1:53086 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20251023 20:12:04.635048 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 12739
I20251023 20:12:04.639776 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 12875
I20251023 20:12:04.644771 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 13011
I20251023 20:12:04.650061 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 12643
2025-10-23T20:12:04Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3085 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[13163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[13163](info): set up 2 sockets
Oct 23 20:12:04 dist-test-slave-x62p krb5kdc[13163](info): commencing operation
krb5kdc: starting...
W20251023 20:12:06.649108 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:06 dist-test-slave-x62p krb5kdc[13163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:06.783882 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:35591
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:35591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:06.865887 13179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:06.866091 13179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:06.866132 13179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:06.867621 13179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:06.867678 13179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:06.867700 13179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:06.867722 13179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:06.867745 13179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:06.869441 13179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:35591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:35591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:06.869736 13179 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:06.869932 13179 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:06.872546 13187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:06.872529 13185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:06.872529 13184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:06.872593 13179 server_base.cc:1047] running on GCE node
I20251023 20:12:06.873055 13179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:06.873303 13179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:06.874449 13179 hybrid_clock.cc:648] HybridClock initialized: now 1761250326874433 us; error 29 us; skew 500 ppm
Oct 23 20:12:06 dist-test-slave-x62p krb5kdc[13163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250326, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:06.876667 13179 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:06.877264 13179 webserver.cc:492] Webserver started at http://127.9.117.254:37037/ using document root <none> and password file <none>
I20251023 20:12:06.877467 13179 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:06.877506 13179 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:06.877616 13179 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:06.878613 13179 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "00960146382c4108871ed5142551c840"
format_stamp: "Formatted at 2025-10-23 20:12:06 on dist-test-slave-x62p"
server_key: "a91179b9c897aa1d42811a64b03150e1"
server_key_iv: "ac20f538c8918e2e0169ddb9e4e04169"
server_key_version: "encryptionkey@0"
I20251023 20:12:06.878928 13179 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "00960146382c4108871ed5142551c840"
format_stamp: "Formatted at 2025-10-23 20:12:06 on dist-test-slave-x62p"
server_key: "a91179b9c897aa1d42811a64b03150e1"
server_key_iv: "ac20f538c8918e2e0169ddb9e4e04169"
server_key_version: "encryptionkey@0"
I20251023 20:12:06.880146 13179 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:12:06.880844 13194 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:06.881032 13179 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:06.881114 13179 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "00960146382c4108871ed5142551c840"
format_stamp: "Formatted at 2025-10-23 20:12:06 on dist-test-slave-x62p"
server_key: "a91179b9c897aa1d42811a64b03150e1"
server_key_iv: "ac20f538c8918e2e0169ddb9e4e04169"
server_key_version: "encryptionkey@0"
I20251023 20:12:06.881176 13179 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:06.890949 13179 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:06.893077 13179 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:06.893206 13179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:06.897459 13179 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:35591
I20251023 20:12:06.897506 13246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:35591 every 8 connection(s)
I20251023 20:12:06.897811 13179 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:06.898304 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 13179
I20251023 20:12:06.898414 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:06.898505 13247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:06.898624 9687 external_mini_cluster.cc:1468] Setting key 833b5393e2bd803768ab304e9a1b7acb
I20251023 20:12:06.901664 13247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840: Bootstrap starting.
I20251023 20:12:06.902302 13247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:06.902581 13247 log.cc:826] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:06.903353 13247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840: No bootstrap required, opened a new log
Oct 23 20:12:06 dist-test-slave-x62p krb5kdc[13163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:06.904920 13247 raft_consensus.cc:359] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00960146382c4108871ed5142551c840" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35591 } }
I20251023 20:12:06.905083 13247 raft_consensus.cc:385] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:06.905110 13247 raft_consensus.cc:740] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00960146382c4108871ed5142551c840, State: Initialized, Role: FOLLOWER
I20251023 20:12:06.905205 13247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00960146382c4108871ed5142551c840" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35591 } }
I20251023 20:12:06.905273 13247 raft_consensus.cc:399] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:06.905299 13247 raft_consensus.cc:493] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:06.905346 13247 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:06.906091 13247 raft_consensus.cc:515] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00960146382c4108871ed5142551c840" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35591 } }
I20251023 20:12:06.906217 13247 leader_election.cc:304] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00960146382c4108871ed5142551c840; no voters:
I20251023 20:12:06.906399 13247 leader_election.cc:290] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:06.906621 13252 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:06.906687 13247 sys_catalog.cc:565] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:06.906812 13252 raft_consensus.cc:697] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [term 1 LEADER]: Becoming Leader. State: Replica: 00960146382c4108871ed5142551c840, State: Running, Role: LEADER
I20251023 20:12:06.906936 13252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00960146382c4108871ed5142551c840" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35591 } }
I20251023 20:12:06.907292 13252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00960146382c4108871ed5142551c840. Latest consensus state: current_term: 1 leader_uuid: "00960146382c4108871ed5142551c840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00960146382c4108871ed5142551c840" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35591 } } }
I20251023 20:12:06.907591 13252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:06.907477 13253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00960146382c4108871ed5142551c840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00960146382c4108871ed5142551c840" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35591 } } }
I20251023 20:12:06.908022 13253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:06.908138 13261 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:06.908667 13261 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:06.908978 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:06.899162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.9.117.254:35591)
1023 20:12:06.899310 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:06.899326 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:06.899362 (+ 36us) server_negotiation.cc:408] Connection header received
1023 20:12:06.899735 (+ 373us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:06.899740 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:06.899794 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:06.899907 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:06.900952 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:06.901724 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:06.902318 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:06.902594 (+ 276us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:06.905071 (+ 2477us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:06.905080 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:06.905080 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:06.905112 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:06.907260 (+ 2148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:06.907712 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:06.907715 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:06.907715 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:06.907790 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:06.907987 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:06.907988 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:06.907988 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:06.908322 (+ 334us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:06.908552 (+ 230us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:06.908631 (+ 79us) server_negotiation.cc:300] Negotiation successful
1023 20:12:06.908784 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20251023 20:12:06.909829 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:12:06.911948 13261 catalog_manager.cc:1357] Generated new cluster ID: 5df3ad18f9584572b97ad0d85f425762
I20251023 20:12:06.912005 13261 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:06.922947 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:06.919634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.9.117.254:35591)
1023 20:12:06.919698 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:06.919700 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:06.919837 (+ 137us) server_negotiation.cc:408] Connection header received
1023 20:12:06.919912 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:06.919913 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:06.919924 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:06.919968 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:06.920527 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:06.921023 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:06.921574 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:06.921777 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:06.922403 (+ 626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:06.922411 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:06.922412 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:06.922646 (+ 234us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:06.922665 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:06.922671 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:06.922751 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:06.922820 (+ 69us) server_negotiation.cc:300] Negotiation successful
1023 20:12:06.922856 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251023 20:12:06.930744 13261 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:06.931411 13261 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:06.942540 13261 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 00960146382c4108871ed5142551c840: Generated new TSK 0
I20251023 20:12:06.942806 13261 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:12:06.954349 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 13179
2025-10-23T20:12:06Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2300 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:07 dist-test-slave-x62p krb5kdc[13280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:07 dist-test-slave-x62p krb5kdc[13280](info): set up 2 sockets
Oct 23 20:12:07 dist-test-slave-x62p krb5kdc[13280](info): commencing operation
krb5kdc: starting...
W20251023 20:12:08.945140 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.973s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:09.089493 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:46855
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:09.173770 13296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:09.173975 13296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:09.174017 13296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:09.175518 13296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:09.175575 13296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:09.175598 13296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:09.175621 13296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:09.175637 13296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:09.177330 13296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:46855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:46855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:09.177637 13296 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:09.177839 13296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:09.180869 13304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:09.180974 13302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:09.180930 13301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:09.181109 13296 server_base.cc:1047] running on GCE node
I20251023 20:12:09.181352 13296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:09.181619 13296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:09.182775 13296 hybrid_clock.cc:648] HybridClock initialized: now 1761250329182756 us; error 37 us; skew 500 ppm
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:09.185355 13296 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:09.185976 13296 webserver.cc:492] Webserver started at http://127.9.117.254:40775/ using document root <none> and password file <none>
I20251023 20:12:09.186164 13296 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:09.186206 13296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:09.186326 13296 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:09.187346 13296 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "75cffca46b804dd28a374c64427a7491"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "dbc326f7346e4de57d8bb75db84aebfc"
server_key_iv: "0576f0793c4031c65d5728252b62ab06"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.187673 13296 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "75cffca46b804dd28a374c64427a7491"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "dbc326f7346e4de57d8bb75db84aebfc"
server_key_iv: "0576f0793c4031c65d5728252b62ab06"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.189047 13296 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251023 20:12:09.189814 13311 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.190016 13296 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:09.190093 13296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "75cffca46b804dd28a374c64427a7491"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "dbc326f7346e4de57d8bb75db84aebfc"
server_key_iv: "0576f0793c4031c65d5728252b62ab06"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.190148 13296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:09.203032 13296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:09.211081 13296 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:09.211205 13296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:09.215426 13296 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:46855
I20251023 20:12:09.215457 13363 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:46855 every 8 connection(s)
I20251023 20:12:09.215806 13296 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:09.216480 13364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:09.219218 13364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491: Bootstrap starting.
I20251023 20:12:09.219832 13364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:09.220079 13364 log.cc:826] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:09.220908 13364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491: No bootstrap required, opened a new log
I20251023 20:12:09.222184 13364 raft_consensus.cc:359] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cffca46b804dd28a374c64427a7491" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46855 } }
I20251023 20:12:09.222313 13364 raft_consensus.cc:385] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:09.222334 13364 raft_consensus.cc:740] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cffca46b804dd28a374c64427a7491, State: Initialized, Role: FOLLOWER
I20251023 20:12:09.222451 13364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cffca46b804dd28a374c64427a7491" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46855 } }
I20251023 20:12:09.222512 13364 raft_consensus.cc:399] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:09.222543 13364 raft_consensus.cc:493] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:09.222591 13364 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:09.223157 13364 raft_consensus.cc:515] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cffca46b804dd28a374c64427a7491" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46855 } }
I20251023 20:12:09.223259 13364 leader_election.cc:304] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75cffca46b804dd28a374c64427a7491; no voters:
I20251023 20:12:09.223420 13364 leader_election.cc:290] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:09.223501 13367 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:09.223652 13364 sys_catalog.cc:565] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:09.223687 13367 raft_consensus.cc:697] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [term 1 LEADER]: Becoming Leader. State: Replica: 75cffca46b804dd28a374c64427a7491, State: Running, Role: LEADER
I20251023 20:12:09.223778 13367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cffca46b804dd28a374c64427a7491" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46855 } }
I20251023 20:12:09.224125 13368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75cffca46b804dd28a374c64427a7491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cffca46b804dd28a374c64427a7491" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46855 } } }
I20251023 20:12:09.224159 13369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75cffca46b804dd28a374c64427a7491. Latest consensus state: current_term: 1 leader_uuid: "75cffca46b804dd28a374c64427a7491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cffca46b804dd28a374c64427a7491" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 46855 } } }
I20251023 20:12:09.224216 13368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:09.224220 13369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:09.224419 13373 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:09.224423 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 13296
I20251023 20:12:09.224519 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:09.224706 9687 external_mini_cluster.cc:1468] Setting key f1e90cdd1e4467cf57a19d779260c1d6
I20251023 20:12:09.225042 13373 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:09.226765 13373 catalog_manager.cc:1357] Generated new cluster ID: a99b2c8c12cd4db2a27504e1809d710d
I20251023 20:12:09.226817 13373 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:09.234146 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:09.225546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.9.117.254:46855)
1023 20:12:09.225713 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:09.225726 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:09.225747 (+ 21us) server_negotiation.cc:408] Connection header received
1023 20:12:09.225878 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:09.225885 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:09.225941 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:09.226112 (+ 171us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:09.227038 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.227793 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:09.228342 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.228600 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:09.230739 (+ 2139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:09.230746 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:09.230747 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:09.230769 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:09.232687 (+ 1918us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.233101 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.233104 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.233104 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.233183 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.233333 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.233334 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.233334 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.233560 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:09.233711 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:09.233812 (+ 101us) server_negotiation.cc:300] Negotiation successful
1023 20:12:09.233952 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20251023 20:12:09.239651 13373 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:09.240206 13373 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:09.243753 13373 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 75cffca46b804dd28a374c64427a7491: Generated new TSK 0
I20251023 20:12:09.243933 13373 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:09.305064 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46855
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:09.394737 13392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:09.394994 13392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:09.395043 13392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:09.397584 13392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:09.397657 13392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:09.397693 13392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:09.399683 13392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:09.400024 13392 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:09.400243 13392 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:09.403090 13397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:09.403499 13398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:09.403679 13392 server_base.cc:1047] running on GCE node
W20251023 20:12:09.403582 13400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:09.404011 13392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:09.404276 13392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:09.405460 13392 hybrid_clock.cc:648] HybridClock initialized: now 1761250329405444 us; error 32 us; skew 500 ppm
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:09.408190 13392 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:12:09.408843 13392 webserver.cc:492] Webserver started at http://127.9.117.193:37013/ using document root <none> and password file <none>
I20251023 20:12:09.409052 13392 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:09.409096 13392 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:09.409224 13392 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:09.410323 13392 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "81362ea5ffb141918669841e476768f2"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "f05960722b86c6072c4c6ee173e2550b"
server_key_iv: "17f0f6477841386ebc44745f6f1c7891"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.410684 13392 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "81362ea5ffb141918669841e476768f2"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "f05960722b86c6072c4c6ee173e2550b"
server_key_iv: "17f0f6477841386ebc44745f6f1c7891"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.412091 13392 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:12:09.412968 13407 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.413172 13392 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:09.413260 13392 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "81362ea5ffb141918669841e476768f2"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "f05960722b86c6072c4c6ee173e2550b"
server_key_iv: "17f0f6477841386ebc44745f6f1c7891"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.413333 13392 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:09.424521 13392 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:09.426988 13392 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:09.427147 13392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:09.427389 13392 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:09.427731 13392 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:09.427776 13392 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.427814 13392 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:09.427843 13392 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.434366 13392 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:36951
I20251023 20:12:09.434464 13520 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:36951 every 8 connection(s)
I20251023 20:12:09.434780 13392 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:09.440558 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 13392
I20251023 20:12:09.440689 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:09.440939 9687 external_mini_cluster.cc:1468] Setting key da734a5801acec2d066644cb59c87f21
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:09.444861 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:09.435588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:60815 (local address 127.9.117.254:46855)
1023 20:12:09.435693 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:09.435695 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:09.435837 (+ 142us) server_negotiation.cc:408] Connection header received
1023 20:12:09.436392 (+ 555us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:09.436393 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:09.436413 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:09.436500 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:09.437400 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.437932 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:09.438515 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.438700 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:09.441686 (+ 2986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:09.441690 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:09.441690 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:09.441699 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:09.443412 (+ 1713us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.443891 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.443892 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.443893 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.443924 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.444122 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.444123 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.444123 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.444236 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:09.444301 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:09.444600 (+ 299us) server_negotiation.cc:300] Negotiation successful
1023 20:12:09.444749 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251023 20:12:09.444988 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:09.435473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46855 (local address 127.9.117.193:60815)
1023 20:12:09.435694 (+ 221us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:09.435749 (+ 55us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:09.436291 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:09.436555 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:09.436563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:09.436971 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:09.437286 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:09.437290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.438029 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:09.438030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:09.438429 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:09.438431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.438581 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:09.439620 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:09.439640 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:09.441587 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:09.443551 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:09.443563 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:09.443567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:09.443810 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:09.443984 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:09.443985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:09.443986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:09.444060 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:09.444294 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:09.444300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:09.444451 (+ 151us) client_negotiation.cc:769] Sending connection context
1023 20:12:09.444568 (+ 117us) client_negotiation.cc:240] Negotiation successful
1023 20:12:09.444758 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
I20251023 20:12:09.445598 13521 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46855
I20251023 20:12:09.445701 13521 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:09.445940 13521 heartbeater.cc:507] Master 127.9.117.254:46855 requested a full tablet report, sending...
I20251023 20:12:09.446543 13328 ts_manager.cc:194] Registered new tserver with Master: 81362ea5ffb141918669841e476768f2 (127.9.117.193:36951)
I20251023 20:12:09.447669 13328 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:60815
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:09.489382 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46855
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:09.574071 13528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:09.574275 13528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:09.574303 13528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:09.575770 13528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:09.575816 13528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:09.575845 13528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:09.577476 13528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:09.577690 13528 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:09.577854 13528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:09.580457 13536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:09.580557 13533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:09.580458 13534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:09.580905 13528 server_base.cc:1047] running on GCE node
I20251023 20:12:09.581073 13528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:09.581274 13528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:09.582418 13528 hybrid_clock.cc:648] HybridClock initialized: now 1761250329582400 us; error 30 us; skew 500 ppm
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:09.585095 13528 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:12:09.585675 13528 webserver.cc:492] Webserver started at http://127.9.117.194:42699/ using document root <none> and password file <none>
I20251023 20:12:09.585879 13528 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:09.585920 13528 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:09.586040 13528 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:09.587047 13528 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "c7c6f944fd2e42228fc813241325076f"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "4b162b459d0e31eb084ed1d3252af018"
server_key_iv: "27dc7c26a6851817d6f80d4722a392d0"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.587376 13528 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "c7c6f944fd2e42228fc813241325076f"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "4b162b459d0e31eb084ed1d3252af018"
server_key_iv: "27dc7c26a6851817d6f80d4722a392d0"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.588609 13528 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:09.589370 13543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.589546 13528 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:09.589622 13528 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "c7c6f944fd2e42228fc813241325076f"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "4b162b459d0e31eb084ed1d3252af018"
server_key_iv: "27dc7c26a6851817d6f80d4722a392d0"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.589689 13528 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:09.600124 13528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:09.602344 13528 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:09.602501 13528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:09.602730 13528 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:09.603055 13528 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:09.603091 13528 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.603128 13528 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:09.603158 13528 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.609236 13528 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:34829
I20251023 20:12:09.609306 13656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:34829 every 8 connection(s)
I20251023 20:12:09.609615 13528 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:12:09.614401 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 13528
I20251023 20:12:09.614506 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:09.614653 9687 external_mini_cluster.cc:1468] Setting key 613c016fb7241bc12264fbf90f00da32
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:09.619014 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:09.610349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:59833 (local address 127.9.117.254:46855)
1023 20:12:09.610431 (+ 82us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:09.610434 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:09.610623 (+ 189us) server_negotiation.cc:408] Connection header received
1023 20:12:09.611161 (+ 538us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:09.611163 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:09.611175 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:09.611236 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:09.612055 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.612563 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:09.613176 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.613343 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:09.615863 (+ 2520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:09.615868 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:09.615869 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:09.615878 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:09.617599 (+ 1721us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.618114 (+ 515us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.618115 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.618115 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.618140 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.618338 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.618338 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.618338 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.618431 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:09.618477 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:09.618856 (+ 379us) server_negotiation.cc:300] Negotiation successful
1023 20:12:09.618944 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251023 20:12:09.619258 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:09.610289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46855 (local address 127.9.117.194:59833)
1023 20:12:09.610478 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:09.610507 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:09.611063 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:09.611300 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:09.611304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:09.611646 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:09.611952 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:09.611956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.612652 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:09.612654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:09.613094 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:09.613096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.613242 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:09.614221 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:09.614239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:09.615770 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:09.617681 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:09.617688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:09.617692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:09.618043 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:09.618200 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:09.618201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:09.618201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:09.618286 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:09.618480 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:09.618485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:09.618685 (+ 200us) client_negotiation.cc:769] Sending connection context
1023 20:12:09.618826 (+ 141us) client_negotiation.cc:240] Negotiation successful
1023 20:12:09.619005 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20251023 20:12:09.619771 13657 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46855
I20251023 20:12:09.619882 13657 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:09.620105 13657 heartbeater.cc:507] Master 127.9.117.254:46855 requested a full tablet report, sending...
I20251023 20:12:09.620496 13328 ts_manager.cc:194] Registered new tserver with Master: c7c6f944fd2e42228fc813241325076f (127.9.117.194:34829)
I20251023 20:12:09.620949 13328 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:59833
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:09.662655 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:46855
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:09.747072 13664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:09.747311 13664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:09.747359 13664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:09.749156 13664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:09.749230 13664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:09.749282 13664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:09.751281 13664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:09.751609 13664 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:09.751830 13664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:09.754778 13670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:09.754779 13672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:09.754778 13669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:09.755125 13664 server_base.cc:1047] running on GCE node
I20251023 20:12:09.755291 13664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:09.755501 13664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:09.756677 13664 hybrid_clock.cc:648] HybridClock initialized: now 1761250329756660 us; error 43 us; skew 500 ppm
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:09.759732 13664 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:12:09.760439 13664 webserver.cc:492] Webserver started at http://127.9.117.195:40933/ using document root <none> and password file <none>
I20251023 20:12:09.760661 13664 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:09.760708 13664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:09.760897 13664 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:09.761981 13664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "5b4b85bebb354a448a7350fc5ba06be5"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "fb9f1cd4e68d5814df98af6071621bc0"
server_key_iv: "7b180de3372cb34ce4b7ebf26a425d67"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.762320 13664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "5b4b85bebb354a448a7350fc5ba06be5"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "fb9f1cd4e68d5814df98af6071621bc0"
server_key_iv: "7b180de3372cb34ce4b7ebf26a425d67"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.763808 13664 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:09.764709 13679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.765009 13664 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:09.765106 13664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "5b4b85bebb354a448a7350fc5ba06be5"
format_stamp: "Formatted at 2025-10-23 20:12:09 on dist-test-slave-x62p"
server_key: "fb9f1cd4e68d5814df98af6071621bc0"
server_key_iv: "7b180de3372cb34ce4b7ebf26a425d67"
server_key_version: "encryptionkey@0"
I20251023 20:12:09.765187 13664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:09.797039 13664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:09.799463 13664 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:09.799669 13664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:09.799932 13664 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:09.800297 13664 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:09.800333 13664 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.800370 13664 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:09.800397 13664 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:09.807260 13664 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:46759
I20251023 20:12:09.807317 13792 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:46759 every 8 connection(s)
I20251023 20:12:09.807665 13664 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:12:09.807897 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 13664
I20251023 20:12:09.807989 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:09.808184 9687 external_mini_cluster.cc:1468] Setting key d1b536fecca7723ef5b2854a5b4831ea
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:09.817479 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:09.808521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:45843 (local address 127.9.117.254:46855)
1023 20:12:09.808592 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:09.808594 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:09.808784 (+ 190us) server_negotiation.cc:408] Connection header received
1023 20:12:09.809464 (+ 680us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:09.809466 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:09.809512 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:09.809576 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:09.810434 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.810975 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:09.811548 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.811658 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:09.814345 (+ 2687us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:09.814349 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:09.814349 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:09.814357 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:09.815898 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.816458 (+ 560us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.816460 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.816460 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.816491 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.816709 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.816710 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.816710 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.816886 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:09.816946 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:09.817302 (+ 356us) server_negotiation.cc:300] Negotiation successful
1023 20:12:09.817404 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251023 20:12:09.817651 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:09.808489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:46855 (local address 127.9.117.195:45843)
1023 20:12:09.808652 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:09.808687 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:09.809348 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:09.809658 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:09.809663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:09.810020 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:09.810324 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:09.810328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.811066 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:09.811067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:09.811468 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:09.811470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.811608 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:09.812625 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:09.812646 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:09.814240 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:09.815993 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:09.816005 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:09.816008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:09.816359 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:09.816583 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:09.816584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:09.816584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:09.816650 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:09.816956 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:09.816963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:09.817113 (+ 150us) client_negotiation.cc:769] Sending connection context
1023 20:12:09.817264 (+ 151us) client_negotiation.cc:240] Negotiation successful
1023 20:12:09.817415 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20251023 20:12:09.818305 13793 heartbeater.cc:344] Connected to a master server at 127.9.117.254:46855
I20251023 20:12:09.818413 13793 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:09.818670 13793 heartbeater.cc:507] Master 127.9.117.254:46855 requested a full tablet report, sending...
I20251023 20:12:09.819087 13328 ts_manager.cc:194] Registered new tserver with Master: 5b4b85bebb354a448a7350fc5ba06be5 (127.9.117.195:46759)
I20251023 20:12:09.819505 13328 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:45843
I20251023 20:12:09.820243 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250329, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:09.843753 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:09.836475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.9.117.254:46855)
1023 20:12:09.836547 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:09.836549 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:09.836630 (+ 81us) server_negotiation.cc:408] Connection header received
1023 20:12:09.836719 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:09.836720 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:09.836760 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:09.836816 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:09.837593 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.838088 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:09.838655 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:09.838855 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:09.841056 (+ 2201us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:09.841061 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:09.841062 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:09.841070 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:09.842541 (+ 1471us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.843005 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.843006 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.843007 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.843037 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:09.843182 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:09.843183 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:09.843183 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:09.843319 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:09.843424 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:09.843536 (+ 112us) server_negotiation.cc:300] Negotiation successful
1023 20:12:09.843638 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20251023 20:12:09.844039 13327 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43556
I20251023 20:12:09.844605 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 13392
I20251023 20:12:09.850085 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 13528
I20251023 20:12:09.854997 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 13664
I20251023 20:12:09.859206 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 13296
2025-10-23T20:12:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2907 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13805](info): set up 2 sockets
Oct 23 20:12:09 dist-test-slave-x62p krb5kdc[13805](info): commencing operation
krb5kdc: starting...
W20251023 20:12:11.861205 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.981s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:11 dist-test-slave-x62p krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:11.997237 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43863
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:12.079301 13821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:12.079510 13821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:12.079551 13821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:12.081118 13821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:12.081175 13821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:12.081202 13821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:12.081221 13821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:12.081241 13821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:12.082917 13821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:43863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:12.083194 13821 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:12.083391 13821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:12.086063 13826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:12.086077 13827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:12.086063 13829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:12.086162 13821 server_base.cc:1047] running on GCE node
I20251023 20:12:12.086463 13821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:12.086711 13821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:12.087848 13821 hybrid_clock.cc:648] HybridClock initialized: now 1761250332087829 us; error 34 us; skew 500 ppm
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:12.090153 13821 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:12.090730 13821 webserver.cc:492] Webserver started at http://127.9.117.254:35791/ using document root <none> and password file <none>
I20251023 20:12:12.090916 13821 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:12.090953 13821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:12.091058 13821 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:12.092079 13821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "c66e41d3348b4b5181856ccbae0c34f8"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "553b6bdcd96bdc94d49be160e1df5e07"
server_key_iv: "a43bb4d590b8be883042467bd80b7c12"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.092402 13821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "c66e41d3348b4b5181856ccbae0c34f8"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "553b6bdcd96bdc94d49be160e1df5e07"
server_key_iv: "a43bb4d590b8be883042467bd80b7c12"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.093871 13821 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:12.094707 13836 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.094907 13821 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:12.094976 13821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "c66e41d3348b4b5181856ccbae0c34f8"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "553b6bdcd96bdc94d49be160e1df5e07"
server_key_iv: "a43bb4d590b8be883042467bd80b7c12"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.095052 13821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:12.104297 13821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:12.106398 13821 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:12.106628 13821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:12.111043 13821 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:43863
I20251023 20:12:12.111070 13888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:43863 every 8 connection(s)
I20251023 20:12:12.111393 13821 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:12.112063 13889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:12.112116 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 13821
I20251023 20:12:12.112211 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:12.112399 9687 external_mini_cluster.cc:1468] Setting key 7f1141f6f341f6befeb1cb4acbf5742d
I20251023 20:12:12.115720 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8: Bootstrap starting.
I20251023 20:12:12.116422 13889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:12.116719 13889 log.cc:826] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:12.117622 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8: No bootstrap required, opened a new log
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:12.118767 13889 raft_consensus.cc:359] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66e41d3348b4b5181856ccbae0c34f8" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43863 } }
I20251023 20:12:12.118888 13889 raft_consensus.cc:385] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:12.118909 13889 raft_consensus.cc:740] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66e41d3348b4b5181856ccbae0c34f8, State: Initialized, Role: FOLLOWER
I20251023 20:12:12.119027 13889 consensus_queue.cc:260] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66e41d3348b4b5181856ccbae0c34f8" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43863 } }
I20251023 20:12:12.119095 13889 raft_consensus.cc:399] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:12.119138 13889 raft_consensus.cc:493] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:12.119186 13889 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:12.119853 13889 raft_consensus.cc:515] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66e41d3348b4b5181856ccbae0c34f8" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43863 } }
I20251023 20:12:12.120000 13889 leader_election.cc:304] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66e41d3348b4b5181856ccbae0c34f8; no voters:
I20251023 20:12:12.120180 13889 leader_election.cc:290] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:12.120246 13894 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:12.120415 13889 sys_catalog.cc:565] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:12.120482 13894 raft_consensus.cc:697] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [term 1 LEADER]: Becoming Leader. State: Replica: c66e41d3348b4b5181856ccbae0c34f8, State: Running, Role: LEADER
I20251023 20:12:12.120644 13894 consensus_queue.cc:237] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66e41d3348b4b5181856ccbae0c34f8" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43863 } }
I20251023 20:12:12.122202 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c66e41d3348b4b5181856ccbae0c34f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66e41d3348b4b5181856ccbae0c34f8" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43863 } } }
I20251023 20:12:12.122407 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:12.122648 13896 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c66e41d3348b4b5181856ccbae0c34f8. Latest consensus state: current_term: 1 leader_uuid: "c66e41d3348b4b5181856ccbae0c34f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66e41d3348b4b5181856ccbae0c34f8" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43863 } } }
W20251023 20:12:12.122686 13909 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20251023 20:12:12.122792 13896 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8 [sys.catalog]: This master's current role is: LEADER
W20251023 20:12:12.122817 13909 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:12:12.122918 13911 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:12.123091 13911 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:12.123600 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.113057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.9.117.254:43863)
1023 20:12:12.113223 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:12.113233 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:12.113255 (+ 22us) server_negotiation.cc:408] Connection header received
1023 20:12:12.113989 (+ 734us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:12.113996 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:12.114049 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:12.114166 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:12.115253 (+ 1087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.116118 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.116931 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.117225 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.119450 (+ 2225us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:12.119458 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:12.119459 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:12.119479 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:12.122001 (+ 2522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.122456 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.122459 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.122460 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.122543 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.122743 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.122744 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.122744 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.123004 (+ 260us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:12.123157 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:12.123273 (+ 116us) server_negotiation.cc:300] Negotiation successful
1023 20:12:12.123379 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20251023 20:12:12.124573 13911 catalog_manager.cc:1357] Generated new cluster ID: 3c6ba4b2037d4b529bdcd68e5d18c01c
I20251023 20:12:12.124629 13911 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:12.161356 13911 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:12.162195 13911 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:12.169153 13911 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c66e41d3348b4b5181856ccbae0c34f8: Generated new TSK 0
I20251023 20:12:12.169389 13911 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:12.225291 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43863
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:12.307715 13917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:12.307884 13917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:12.307911 13917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:12.309433 13917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:12.309476 13917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:12.309516 13917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:12.311046 13917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:12.311276 13917 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:12.311476 13917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:12.314127 13923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:12.314138 13925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:12.314294 13922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:12.314301 13917 server_base.cc:1047] running on GCE node
I20251023 20:12:12.314584 13917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:12.314779 13917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:12.315908 13917 hybrid_clock.cc:648] HybridClock initialized: now 1761250332315896 us; error 30 us; skew 500 ppm
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:12.318236 13917 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:12:12.318794 13917 webserver.cc:492] Webserver started at http://127.9.117.193:45847/ using document root <none> and password file <none>
I20251023 20:12:12.318994 13917 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:12.319031 13917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:12.319146 13917 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:12.320118 13917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "5fd6c9f3ab434f368f2892b1191394e5"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "c0e3ef73715e28303768390372b1160d"
server_key_iv: "16d8f78ad7efad381021bf11c39add2d"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.320423 13917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "5fd6c9f3ab434f368f2892b1191394e5"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "c0e3ef73715e28303768390372b1160d"
server_key_iv: "16d8f78ad7efad381021bf11c39add2d"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.321885 13917 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:12.322584 13932 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.322796 13917 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:12.322868 13917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "5fd6c9f3ab434f368f2892b1191394e5"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "c0e3ef73715e28303768390372b1160d"
server_key_iv: "16d8f78ad7efad381021bf11c39add2d"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.322919 13917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:12.330761 13917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:12.332762 13917 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:12.332885 13917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:12.333067 13917 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:12.333328 13917 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:12.333356 13917 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.333382 13917 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:12.333400 13917 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.339218 13917 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:39851
I20251023 20:12:12.339289 14045 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:39851 every 8 connection(s)
I20251023 20:12:12.339565 13917 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:12.339761 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 13917
I20251023 20:12:12.339865 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:12.340035 9687 external_mini_cluster.cc:1468] Setting key eac9c5595b74021a1d421329589b3c27
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:12:12.349279 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.340327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:54851 (local address 127.9.117.254:43863)
1023 20:12:12.340383 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:12.340386 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:12.340578 (+ 192us) server_negotiation.cc:408] Connection header received
1023 20:12:12.341282 (+ 704us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:12.341283 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:12.341292 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:12.341336 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:12.342442 (+ 1106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.343255 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.344306 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.344512 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.346363 (+ 1851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:12.346371 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:12.346371 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:12.346382 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:12.348014 (+ 1632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.348432 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.348434 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.348435 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.348467 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.348631 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.348631 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.348631 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.348741 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:12.348814 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:12.349097 (+ 283us) server_negotiation.cc:300] Negotiation successful
1023 20:12:12.349201 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251023 20:12:12.349453 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.340272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43863 (local address 127.9.117.193:54851)
1023 20:12:12.340481 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:12.340503 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:12.341187 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:12.341399 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:12.341403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:12.341913 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:12.342343 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:12.342348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.343348 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.343349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:12.344216 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:12.344219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.344336 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.344898 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:12.344915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:12.346266 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:12.348109 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:12.348116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:12.348118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:12.348356 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:12.348529 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:12.348529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:12.348530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:12.348585 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:12.348817 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:12.348823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:12.348964 (+ 141us) client_negotiation.cc:769] Sending connection context
1023 20:12:12.349075 (+ 111us) client_negotiation.cc:240] Negotiation successful
1023 20:12:12.349197 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20251023 20:12:12.350068 14046 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43863
I20251023 20:12:12.350178 14046 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:12.350399 14046 heartbeater.cc:507] Master 127.9.117.254:43863 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
I20251023 20:12:12.350947 13853 ts_manager.cc:194] Registered new tserver with Master: 5fd6c9f3ab434f368f2892b1191394e5 (127.9.117.193:39851)
I20251023 20:12:12.351933 13853 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:54851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:12.387741 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43863
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:12.469084 14053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:12.469274 14053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:12.469313 14053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:12.470762 14053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:12.470813 14053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:12.470856 14053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:12.472441 14053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:12.472751 14053 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:12.472954 14053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:12.475521 14061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:12.475543 14058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:12.475587 14059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:12.475723 14053 server_base.cc:1047] running on GCE node
I20251023 20:12:12.475920 14053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:12.476125 14053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:12.477268 14053 hybrid_clock.cc:648] HybridClock initialized: now 1761250332477254 us; error 28 us; skew 500 ppm
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:12.479660 14053 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:12:12.480194 14053 webserver.cc:492] Webserver started at http://127.9.117.194:39083/ using document root <none> and password file <none>
I20251023 20:12:12.480369 14053 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:12.480401 14053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:12.480494 14053 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:12.481487 14053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "ab987082331f4998a43834afcf4b9ba2"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "82562b2bf08ed52820d978511de1915a"
server_key_iv: "83dff1c65d7843ae2dc638595b90a58c"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.481763 14053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "ab987082331f4998a43834afcf4b9ba2"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "82562b2bf08ed52820d978511de1915a"
server_key_iv: "83dff1c65d7843ae2dc638595b90a58c"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.483011 14053 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:12:12.483625 14068 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.483817 14053 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:12:12.483888 14053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "ab987082331f4998a43834afcf4b9ba2"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "82562b2bf08ed52820d978511de1915a"
server_key_iv: "83dff1c65d7843ae2dc638595b90a58c"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.483950 14053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:12.496588 14053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:12.498415 14053 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:12.498522 14053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:12.498706 14053 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:12.499037 14053 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:12.499070 14053 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.499094 14053 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:12.499105 14053 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.504930 14053 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:41115
I20251023 20:12:12.505007 14181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:41115 every 8 connection(s)
I20251023 20:12:12.505280 14053 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:12.512733 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 14053
I20251023 20:12:12.512871 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:12.513051 9687 external_mini_cluster.cc:1468] Setting key a87c0101daa4ff020af3527b37cbbb70
I20251023 20:12:12.513964 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.505975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:41893 (local address 127.9.117.254:43863)
1023 20:12:12.506053 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:12.506056 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:12.506245 (+ 189us) server_negotiation.cc:408] Connection header received
1023 20:12:12.506746 (+ 501us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:12.506747 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:12.506760 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:12.506813 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:12.507570 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.508072 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.509119 (+ 1047us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.509304 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.511206 (+ 1902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:12.511212 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:12.511212 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:12.511220 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:12.512434 (+ 1214us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.512977 (+ 543us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.512978 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.512979 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.513009 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.513259 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.513260 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.513260 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.513385 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:12.513455 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:12.513749 (+ 294us) server_negotiation.cc:300] Negotiation successful
1023 20:12:12.513846 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251023 20:12:12.514020 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.505920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43863 (local address 127.9.117.194:41893)
1023 20:12:12.506128 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:12.506152 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:12.506658 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:12.506884 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:12.506888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:12.507192 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:12.507477 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:12.507480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.508158 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.508160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:12.509035 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:12.509037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.509172 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.509728 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:12.509745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:12.511124 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:12.512523 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:12.512529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:12.512532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:12.512888 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:12.513097 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:12.513098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:12.513099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:12.513185 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:12.513456 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:12.513463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:12.513600 (+ 137us) client_negotiation.cc:769] Sending connection context
1023 20:12:12.513727 (+ 127us) client_negotiation.cc:240] Negotiation successful
1023 20:12:12.513871 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20251023 20:12:12.514704 14182 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43863
I20251023 20:12:12.514818 14182 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:12.515045 14182 heartbeater.cc:507] Master 127.9.117.254:43863 requested a full tablet report, sending...
I20251023 20:12:12.515430 13853 ts_manager.cc:194] Registered new tserver with Master: ab987082331f4998a43834afcf4b9ba2 (127.9.117.194:41115)
I20251023 20:12:12.515964 13853 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:41893
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:12.557911 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43863
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:12.639827 14189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:12.639997 14189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:12.640022 14189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:12.641496 14189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:12.641544 14189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:12.641579 14189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:12.643127 14189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:12.643360 14189 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:12.643563 14189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:12.646065 14197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:12.646065 14194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:12.646170 14195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:12.646202 14189 server_base.cc:1047] running on GCE node
I20251023 20:12:12.646447 14189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:12.646637 14189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:12.647765 14189 hybrid_clock.cc:648] HybridClock initialized: now 1761250332647757 us; error 27 us; skew 500 ppm
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:12.650174 14189 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:12:12.650756 14189 webserver.cc:492] Webserver started at http://127.9.117.195:43115/ using document root <none> and password file <none>
I20251023 20:12:12.650940 14189 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:12.650975 14189 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:12.651090 14189 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:12.652081 14189 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "d21d119639334026a0e620cd8015eeb7"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "72dd24ffb72190122ad0b364bae2cfaf"
server_key_iv: "1c88d7740e82b27f356b542b896529d1"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.652388 14189 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "d21d119639334026a0e620cd8015eeb7"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "72dd24ffb72190122ad0b364bae2cfaf"
server_key_iv: "1c88d7740e82b27f356b542b896529d1"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.653630 14189 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:12:12.654282 14204 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.654467 14189 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:12.654544 14189 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "d21d119639334026a0e620cd8015eeb7"
format_stamp: "Formatted at 2025-10-23 20:12:12 on dist-test-slave-x62p"
server_key: "72dd24ffb72190122ad0b364bae2cfaf"
server_key_iv: "1c88d7740e82b27f356b542b896529d1"
server_key_version: "encryptionkey@0"
I20251023 20:12:12.654618 14189 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:12.669953 14189 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:12.671988 14189 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:12.672128 14189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:12.672339 14189 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:12.672657 14189 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:12.672690 14189 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.672760 14189 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:12.672788 14189 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:12.678962 14189 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:41869
I20251023 20:12:12.679013 14317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:41869 every 8 connection(s)
I20251023 20:12:12.679309 14189 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:12:12.682279 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 14189
I20251023 20:12:12.682360 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:12.682495 9687 external_mini_cluster.cc:1468] Setting key 58f70ed59d0bba3800fa994e90c8e585
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:12.687611 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.680010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:38021 (local address 127.9.117.254:43863)
1023 20:12:12.680062 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:12.680065 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:12.680235 (+ 170us) server_negotiation.cc:408] Connection header received
1023 20:12:12.680808 (+ 573us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:12.680810 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:12.680822 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:12.680870 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:12.681610 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.682110 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.683110 (+ 1000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.683294 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.685191 (+ 1897us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:12.685195 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:12.685195 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:12.685201 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:12.686444 (+ 1243us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.686809 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.686811 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.686811 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.686839 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.687009 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.687010 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.687011 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.687103 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:12.687164 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:12.687412 (+ 248us) server_negotiation.cc:300] Negotiation successful
1023 20:12:12.687509 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:12:12.687760 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.679956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43863 (local address 127.9.117.195:38021)
1023 20:12:12.680129 (+ 173us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:12.680158 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:12.680689 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:12.680931 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:12.680934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:12.681235 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:12.681516 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:12.681519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.682194 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.682195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:12.683028 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:12.683030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.683162 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.683675 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:12.683691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:12.685085 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:12.686530 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:12.686536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:12.686539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:12.686741 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:12.686895 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:12.686896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:12.686896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:12.686955 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:12.687158 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:12.687163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:12.687289 (+ 126us) client_negotiation.cc:769] Sending connection context
1023 20:12:12.687388 (+ 99us) client_negotiation.cc:240] Negotiation successful
1023 20:12:12.687564 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20251023 20:12:12.688169 14318 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43863
I20251023 20:12:12.688251 14318 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:12.688421 14318 heartbeater.cc:507] Master 127.9.117.254:43863 requested a full tablet report, sending...
I20251023 20:12:12.688787 13853 ts_manager.cc:194] Registered new tserver with Master: d21d119639334026a0e620cd8015eeb7 (127.9.117.195:41869)
I20251023 20:12:12.689131 13853 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:38021
I20251023 20:12:12.694140 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:12:12.715927 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.708532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.9.117.193:39851)
1023 20:12:12.708709 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:12.708717 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:12.708753 (+ 36us) server_negotiation.cc:408] Connection header received
1023 20:12:12.708790 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:12.708792 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:12.708807 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:12.708883 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:12.709587 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.710369 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.710942 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.711100 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.713333 (+ 2233us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:12.713345 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:12.713345 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:12.713352 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:12.714879 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.715208 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.715211 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.715212 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.715248 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.715388 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.715389 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.715390 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.715501 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:12.715654 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:12.715699 (+ 45us) server_negotiation.cc:300] Negotiation successful
1023 20:12:12.715799 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:12.723191 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:12.717204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.9.117.254:43863)
1023 20:12:12.717267 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:12.717270 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:12.717279 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:12.717358 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:12.717359 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:12.717372 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:12.717414 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:12.718017 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.718499 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:12.719013 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:12.719193 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:12.721270 (+ 2077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:12.721274 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:12.721274 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:12.721281 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:12.722452 (+ 1171us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.722713 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.722713 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.722714 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.722729 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:12.722821 (+ 92us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:12.722822 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:12.722822 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:12.722931 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:12.723002 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:12.723065 (+ 63us) server_negotiation.cc:300] Negotiation successful
1023 20:12:12.723124 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20251023 20:12:12.723431 13853 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56408
I20251023 20:12:12.723773 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 13917
I20251023 20:12:12.729377 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 14053
I20251023 20:12:12.734107 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 14189
I20251023 20:12:12.738545 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 13821
2025-10-23T20:12:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2878 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[14330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[14330](info): set up 2 sockets
Oct 23 20:12:12 dist-test-slave-x62p krb5kdc[14330](info): commencing operation
krb5kdc: starting...
W20251023 20:12:14.733358 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:14 dist-test-slave-x62p krb5kdc[14330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:14.873237 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:42395
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:14.956175 14346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:14.956370 14346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:14.956413 14346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:14.958002 14346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:14.958062 14346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:14.958091 14346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:14.958109 14346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:14.958129 14346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:14.959867 14346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:42395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:14.960170 14346 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:14.960366 14346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:14.963039 14352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:14.962990 14354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:14.962985 14351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:14.963388 14346 server_base.cc:1047] running on GCE node
I20251023 20:12:14.963587 14346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:14.963801 14346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:14.964953 14346 hybrid_clock.cc:648] HybridClock initialized: now 1761250334964937 us; error 27 us; skew 500 ppm
Oct 23 20:12:14 dist-test-slave-x62p krb5kdc[14330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250334, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:14.967295 14346 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:14.967612 14346 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:14.967662 14346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:14.967793 14346 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:14.968848 14346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "c1f33cd816204460aaf38142ad4c4615"
format_stamp: "Formatted at 2025-10-23 20:12:14 on dist-test-slave-x62p"
server_key: "0b53203ee799110d83a377ec44e55758"
server_key_iv: "6550925a654d379f3cf9932dde743d83"
server_key_version: "encryptionkey@0"
I20251023 20:12:14.969178 14346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "c1f33cd816204460aaf38142ad4c4615"
format_stamp: "Formatted at 2025-10-23 20:12:14 on dist-test-slave-x62p"
server_key: "0b53203ee799110d83a377ec44e55758"
server_key_iv: "6550925a654d379f3cf9932dde743d83"
server_key_version: "encryptionkey@0"
I20251023 20:12:14.970404 14346 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:14.971086 14360 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:14.971248 14346 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:14.971321 14346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "c1f33cd816204460aaf38142ad4c4615"
format_stamp: "Formatted at 2025-10-23 20:12:14 on dist-test-slave-x62p"
server_key: "0b53203ee799110d83a377ec44e55758"
server_key_iv: "6550925a654d379f3cf9932dde743d83"
server_key_version: "encryptionkey@0"
I20251023 20:12:14.971383 14346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:14.979892 14346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:14.985550 14346 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:14.985703 14346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:14.990105 14346 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:42395
I20251023 20:12:14.990154 14412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:42395 every 8 connection(s)
I20251023 20:12:14.990427 14346 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:14.991051 14413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:14.993803 14413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615: Bootstrap starting.
I20251023 20:12:14.994446 14413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:14.994722 14413 log.cc:826] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:14.995467 14413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615: No bootstrap required, opened a new log
I20251023 20:12:14.996824 14413 raft_consensus.cc:359] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f33cd816204460aaf38142ad4c4615" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42395 } }
I20251023 20:12:14.996976 14413 raft_consensus.cc:385] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:14.996999 14413 raft_consensus.cc:740] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f33cd816204460aaf38142ad4c4615, State: Initialized, Role: FOLLOWER
I20251023 20:12:14.997123 14413 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f33cd816204460aaf38142ad4c4615" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42395 } }
I20251023 20:12:14.997191 14413 raft_consensus.cc:399] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:14.997215 14413 raft_consensus.cc:493] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:14.997251 14413 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:14.997856 14413 raft_consensus.cc:515] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f33cd816204460aaf38142ad4c4615" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42395 } }
I20251023 20:12:14.997959 14413 leader_election.cc:304] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1f33cd816204460aaf38142ad4c4615; no voters:
I20251023 20:12:14.998024 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 14346
I20251023 20:12:14.998133 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:14.998152 14413 leader_election.cc:290] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:14.998219 14416 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:14.998332 9687 external_mini_cluster.cc:1468] Setting key 21790a14cdb33b27a9895dc66ecf7d72
I20251023 20:12:14.998389 14416 raft_consensus.cc:697] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [term 1 LEADER]: Becoming Leader. State: Replica: c1f33cd816204460aaf38142ad4c4615, State: Running, Role: LEADER
I20251023 20:12:14.998395 14413 sys_catalog.cc:565] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:14.998648 14416 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f33cd816204460aaf38142ad4c4615" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42395 } }
I20251023 20:12:14.999106 14418 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1f33cd816204460aaf38142ad4c4615. Latest consensus state: current_term: 1 leader_uuid: "c1f33cd816204460aaf38142ad4c4615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f33cd816204460aaf38142ad4c4615" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42395 } } }
I20251023 20:12:14.999207 14418 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:14.999580 14417 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1f33cd816204460aaf38142ad4c4615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f33cd816204460aaf38142ad4c4615" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42395 } } }
I20251023 20:12:14.999651 14417 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:14.999789 14425 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:15.000319 14425 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:15.002039 14425 catalog_manager.cc:1357] Generated new cluster ID: c21daa4605bf48668dc5a27f7c4e93f2
I20251023 20:12:15.002099 14425 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:12:15 dist-test-slave-x62p krb5kdc[14330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:15.009836 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:14.998964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.9.117.254:42395)
1023 20:12:14.999114 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:14.999127 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:14.999154 (+ 27us) server_negotiation.cc:408] Connection header received
1023 20:12:14.999273 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:14.999279 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:14.999333 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:14.999452 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:15.000890 (+ 1438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.002249 (+ 1359us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.003051 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.003294 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.005975 (+ 2681us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:15.005983 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:15.005984 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:15.006018 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:15.008256 (+ 2238us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.008681 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.008683 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.008684 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.008773 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.008938 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.008939 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.008939 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.009199 (+ 260us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:15.009341 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.009475 (+ 134us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.009613 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20251023 20:12:15.011786 14425 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:15.012342 14425 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:15.017108 14425 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615: Generated new TSK 0
I20251023 20:12:15.017299 14425 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:15.075944 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:42395
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:15.166126 14441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:15.166337 14441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:15.166368 14441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:15.168073 14441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:15.168148 14441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:15.168184 14441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:15.170228 14441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:15.170567 14441 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:15.170789 14441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:15.174088 14447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:15.174088 14449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.174196 14441 server_base.cc:1047] running on GCE node
W20251023 20:12:15.174088 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.174540 14441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:15.174777 14441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:15.175935 14441 hybrid_clock.cc:648] HybridClock initialized: now 1761250335175889 us; error 58 us; skew 500 ppm
Oct 23 20:12:15 dist-test-slave-x62p krb5kdc[14330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250335, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:15.178685 14441 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:12:15.179090 14441 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:15.179149 14441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:15.179278 14441 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:15.180373 14441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "3e2990814298403ab5d3de7e774f11c7"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "6f0fa8993bef4a185837e4c9b7602371"
server_key_iv: "691b4defeb97b05980c057bd0ba00de9"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.180823 14441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "3e2990814298403ab5d3de7e774f11c7"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "6f0fa8993bef4a185837e4c9b7602371"
server_key_iv: "691b4defeb97b05980c057bd0ba00de9"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.182221 14441 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:15.183080 14455 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.183364 14441 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:15.183467 14441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "3e2990814298403ab5d3de7e774f11c7"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "6f0fa8993bef4a185837e4c9b7602371"
server_key_iv: "691b4defeb97b05980c057bd0ba00de9"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.183579 14441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:15.198150 14441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:15.200549 14441 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:15.200809 14441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:15.201077 14441 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:15.201426 14441 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:15.201462 14441 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.201499 14441 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:15.201521 14441 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.207793 14441 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:34543
I20251023 20:12:15.207893 14568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:34543 every 8 connection(s)
I20251023 20:12:15.208125 14441 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:15.211024 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 14441
I20251023 20:12:15.211135 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:15.211311 9687 external_mini_cluster.cc:1468] Setting key 452582b311c56032721dcee39d4a095b
Oct 23 20:12:15 dist-test-slave-x62p krb5kdc[14330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250335, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:15.218430 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.209018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:36733 (local address 127.9.117.254:42395)
1023 20:12:15.209081 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.209084 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.209159 (+ 75us) server_negotiation.cc:408] Connection header received
1023 20:12:15.209706 (+ 547us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.209707 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.209723 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.209794 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:15.211293 (+ 1499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.211897 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.212532 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.212706 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.215114 (+ 2408us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:15.215120 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:15.215120 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:15.215130 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:15.216884 (+ 1754us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.217379 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.217381 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.217382 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.217417 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.217643 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.217644 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.217644 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.217775 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:15.217854 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.218184 (+ 330us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.218317 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:12:15.218564 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.208878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:42395 (local address 127.9.117.193:36733)
1023 20:12:15.209042 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.209075 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.209591 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.209913 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.209918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.210304 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:15.211036 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.211041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.212015 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.212016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:15.212437 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.212438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.212581 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.213262 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:15.213303 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:15.214993 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:15.216992 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:15.217005 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:15.217009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:15.217280 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:15.217494 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:15.217495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:15.217495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:15.217576 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:15.217851 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:15.217859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:15.218020 (+ 161us) client_negotiation.cc:769] Sending connection context
1023 20:12:15.218141 (+ 121us) client_negotiation.cc:240] Negotiation successful
1023 20:12:15.218326 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20251023 20:12:15.219149 14569 heartbeater.cc:344] Connected to a master server at 127.9.117.254:42395
I20251023 20:12:15.219269 14569 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:15.219460 14569 heartbeater.cc:507] Master 127.9.117.254:42395 requested a full tablet report, sending...
I20251023 20:12:15.220043 14377 ts_manager.cc:194] Registered new tserver with Master: 3e2990814298403ab5d3de7e774f11c7 (127.9.117.193:34543)
I20251023 20:12:15.220831 14377 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:36733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:15.260396 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:42395
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:15.342859 14576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:15.343045 14576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:15.343072 14576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:15.344540 14576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:15.344606 14576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:15.344635 14576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:15.346825 14576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:15.347102 14576 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:15.347307 14576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:15.349988 14584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:15.349951 14582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.350116 14576 server_base.cc:1047] running on GCE node
W20251023 20:12:15.349956 14581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.350445 14576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:15.350683 14576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:15.351838 14576 hybrid_clock.cc:648] HybridClock initialized: now 1761250335351816 us; error 35 us; skew 500 ppm
Oct 23 20:12:15 dist-test-slave-x62p krb5kdc[14330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250335, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:15.354313 14576 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:12:15.354626 14576 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:15.354668 14576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:15.354766 14576 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:15.355835 14576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "9108577656d1465bba696b56bcf833b2"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "c0428bc65940f56350b62d630d70825c"
server_key_iv: "d6aa2dc78cab66f8d791c2ffbec67513"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.356122 14576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "9108577656d1465bba696b56bcf833b2"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "c0428bc65940f56350b62d630d70825c"
server_key_iv: "d6aa2dc78cab66f8d791c2ffbec67513"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.357427 14576 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20251023 20:12:15.358160 14590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.358326 14576 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:15.358394 14576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "9108577656d1465bba696b56bcf833b2"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "c0428bc65940f56350b62d630d70825c"
server_key_iv: "d6aa2dc78cab66f8d791c2ffbec67513"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.358464 14576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:15.375133 14576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:15.377291 14576 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:15.377450 14576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:15.377701 14576 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:15.378031 14576 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:15.378065 14576 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.378089 14576 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:15.378115 14576 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.385027 14576 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:45243
I20251023 20:12:15.385123 14703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:45243 every 8 connection(s)
I20251023 20:12:15.385356 14576 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:12:15 dist-test-slave-x62p krb5kdc[14330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250335, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:15.394243 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.386145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:47035 (local address 127.9.117.254:42395)
1023 20:12:15.386223 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.386225 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.386438 (+ 213us) server_negotiation.cc:408] Connection header received
1023 20:12:15.386962 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.386963 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.386973 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.387033 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:15.388245 (+ 1212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.388791 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.389402 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.389522 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.391618 (+ 2096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:15.391624 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:15.391624 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:15.391631 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:15.393071 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.393492 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.393494 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.393494 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.393520 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.393680 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.393681 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.393681 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.393777 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:15.393829 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.394087 (+ 258us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.394167 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251023 20:12:15.394394 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.386071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:42395 (local address 127.9.117.194:47035)
1023 20:12:15.386296 (+ 225us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.386324 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.386863 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.387094 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.387098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.387444 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:15.388145 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.388149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.388874 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.388876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:15.389319 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.389321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.389428 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.389997 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:15.390018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:15.391508 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:15.393173 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:15.393180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:15.393182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:15.393414 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:15.393574 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:15.393574 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:15.393575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:15.393637 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:15.393820 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:15.393822 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:15.393961 (+ 139us) client_negotiation.cc:769] Sending connection context
1023 20:12:15.394070 (+ 109us) client_negotiation.cc:240] Negotiation successful
1023 20:12:15.394201 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20251023 20:12:15.395008 14704 heartbeater.cc:344] Connected to a master server at 127.9.117.254:42395
I20251023 20:12:15.395133 14704 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:15.395341 14704 heartbeater.cc:507] Master 127.9.117.254:42395 requested a full tablet report, sending...
I20251023 20:12:15.395655 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 14576
I20251023 20:12:15.395766 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:15.395979 9687 external_mini_cluster.cc:1468] Setting key ea68a1ec736adf497a9c0749275aa876
I20251023 20:12:15.395979 14377 ts_manager.cc:194] Registered new tserver with Master: 9108577656d1465bba696b56bcf833b2 (127.9.117.194:45243)
I20251023 20:12:15.396397 14377 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:47035
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:15.441470 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:42395
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:15.523856 14711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:15.524017 14711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:15.524040 14711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:15.525475 14711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:15.525527 14711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:15.525552 14711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:15.527050 14711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:15.527266 14711 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:15.527436 14711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:15.530212 14717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:15.530215 14716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.530308 14711 server_base.cc:1047] running on GCE node
W20251023 20:12:15.530211 14719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.530547 14711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:15.530742 14711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:15.531883 14711 hybrid_clock.cc:648] HybridClock initialized: now 1761250335531870 us; error 29 us; skew 500 ppm
Oct 23 20:12:15 dist-test-slave-x62p krb5kdc[14330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250335, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:15.534260 14711 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:12:15.534536 14711 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:15.534577 14711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:15.534675 14711 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:15.535678 14711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "562f34df42634626abf96bf8e58a9912"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "a1e2dc6beab32c126db0909cad78644a"
server_key_iv: "642688bdb2a392678f9e86acb8592cb0"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.535974 14711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "562f34df42634626abf96bf8e58a9912"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "a1e2dc6beab32c126db0909cad78644a"
server_key_iv: "642688bdb2a392678f9e86acb8592cb0"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.537197 14711 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:15.537921 14725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.538093 14711 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:15.538152 14711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "562f34df42634626abf96bf8e58a9912"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "a1e2dc6beab32c126db0909cad78644a"
server_key_iv: "642688bdb2a392678f9e86acb8592cb0"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.538226 14711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:15.558035 14711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:15.559917 14711 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:15.560045 14711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:15.560295 14711 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:15.560627 14711 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:15.560662 14711 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.560699 14711 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:15.560762 14711 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.567042 14711 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:36631
I20251023 20:12:15.567098 14838 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:36631 every 8 connection(s)
I20251023 20:12:15.567341 14711 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
Oct 23 20:12:15 dist-test-slave-x62p krb5kdc[14330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250335, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:15.575692 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.568031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:47269 (local address 127.9.117.254:42395)
1023 20:12:15.568088 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.568090 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.568250 (+ 160us) server_negotiation.cc:408] Connection header received
1023 20:12:15.568720 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.568721 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.568760 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.568832 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:15.569930 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.570413 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.570966 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.571074 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.573170 (+ 2096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:15.573175 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:15.573175 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:15.573182 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:15.574413 (+ 1231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.574808 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.574809 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.574809 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.574835 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.575031 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.575032 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.575033 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.575154 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:15.575222 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.575485 (+ 263us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.575595 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:12:15.575837 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.567986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:42395 (local address 127.9.117.195:47269)
1023 20:12:15.568135 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.568169 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.568646 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.568888 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.568893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.569208 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:15.569828 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.569832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.570481 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.570482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:15.570906 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.570908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.571006 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.571558 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:15.571579 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:15.573066 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:15.574499 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:15.574507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:15.574510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:15.574731 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:15.574890 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:15.574891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:15.574891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:15.574969 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:15.575210 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:15.575216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:15.575357 (+ 141us) client_negotiation.cc:769] Sending connection context
1023 20:12:15.575459 (+ 102us) client_negotiation.cc:240] Negotiation successful
1023 20:12:15.575607 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20251023 20:12:15.576200 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 14711
I20251023 20:12:15.576283 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:15.576406 14839 heartbeater.cc:344] Connected to a master server at 127.9.117.254:42395
I20251023 20:12:15.576453 9687 external_mini_cluster.cc:1468] Setting key 8bc8f641c0990638479abab687524e60
I20251023 20:12:15.576515 14839 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:15.576706 14839 heartbeater.cc:507] Master 127.9.117.254:42395 requested a full tablet report, sending...
I20251023 20:12:15.577147 14376 ts_manager.cc:194] Registered new tserver with Master: 562f34df42634626abf96bf8e58a9912 (127.9.117.195:36631)
I20251023 20:12:15.577595 14376 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:47269
I20251023 20:12:15.588307 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:12:15.595166 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.589688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.9.117.254:42395)
1023 20:12:15.589754 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.589758 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.589848 (+ 90us) server_negotiation.cc:408] Connection header received
1023 20:12:15.589914 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.589915 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.589929 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.589979 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:15.590672 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.591166 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.591698 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.591899 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.592966 (+ 1067us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:15.592970 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:15.592970 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:15.592980 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:15.594192 (+ 1212us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.594479 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.594480 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.594481 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.594505 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:15.594648 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:15.594649 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:15.594649 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:15.594766 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:15.594872 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.594969 (+ 97us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.595054 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251023 20:12:15.596515 14376 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:12:15.597916 14376 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:15.603157 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.600199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:45243 (local address 127.0.0.1:43652)
1023 20:12:15.600469 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.600501 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.600598 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.600845 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.600848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.600851 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:15.601170 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.601173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.602096 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.602097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:15.602847 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.602850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.602922 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.602966 (+ 44us) client_negotiation.cc:769] Sending connection context
1023 20:12:15.603021 (+ 55us) client_negotiation.cc:240] Negotiation successful
1023 20:12:15.603074 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":63,"threads_started":1}
I20251023 20:12:15.603601 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.600200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34543 (local address 127.0.0.1:35580)
1023 20:12:15.600517 (+ 317us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.600527 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.600596 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.601020 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.601021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.601023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:15.601209 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.601211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.602465 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.602466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:15.603364 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.603367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.603430 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.603433 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:15.603462 (+ 29us) client_negotiation.cc:240] Negotiation successful
1023 20:12:15.603494 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20251023 20:12:15.603726 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.600200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:36631 (local address 127.0.0.1:57506)
1023 20:12:15.600538 (+ 338us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.600548 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.600608 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.601142 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.601143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.601145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:15.601398 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.601401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.602700 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.602700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:15.603532 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.603534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.603590 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.603593 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:15.603624 (+ 31us) client_negotiation.cc:240] Negotiation successful
1023 20:12:15.603658 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":199,"thread_start_us":78,"threads_started":1}
I20251023 20:12:15.604034 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.600293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.9.117.194:45243)
1023 20:12:15.600649 (+ 356us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.600658 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.600671 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:12:15.600717 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.600720 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.600764 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.600841 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:15.601239 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.602020 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.603370 (+ 1350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.603855 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.603913 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.603940 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.603970 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":94,"threads_started":1}
I20251023 20:12:15.604532 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.600435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.9.117.193:34543)
1023 20:12:15.600770 (+ 335us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.600782 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.600792 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:15.600834 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.600836 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.600852 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.600922 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:15.601287 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.602374 (+ 1087us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.603779 (+ 1405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.604328 (+ 549us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.604380 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.604410 (+ 30us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.604453 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":80,"threads_started":1}
I20251023 20:12:15.604583 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.600374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.9.117.195:36631)
1023 20:12:15.600748 (+ 374us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.600756 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.600766 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:15.600804 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.600806 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.600822 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.601046 (+ 224us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:15.601482 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.602621 (+ 1139us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.603904 (+ 1283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.604397 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.604435 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.604461 (+ 26us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.604500 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":79,"threads_started":1}
I20251023 20:12:15.604578 14638 tablet_service.cc:1505] Processing CreateTablet for tablet c2420c6e108043e9825fd47b5c6380fe (DEFAULT_TABLE table=test-table [id=38b0e399ae564f9f85a6ea9a7b4041c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:15.604969 14638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2420c6e108043e9825fd47b5c6380fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:15.605072 14503 tablet_service.cc:1505] Processing CreateTablet for tablet c2420c6e108043e9825fd47b5c6380fe (DEFAULT_TABLE table=test-table [id=38b0e399ae564f9f85a6ea9a7b4041c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:15.605334 14503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2420c6e108043e9825fd47b5c6380fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:15.605252 14773 tablet_service.cc:1505] Processing CreateTablet for tablet c2420c6e108043e9825fd47b5c6380fe (DEFAULT_TABLE table=test-table [id=38b0e399ae564f9f85a6ea9a7b4041c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:15.605509 14773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2420c6e108043e9825fd47b5c6380fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:15.607471 14856 tablet_bootstrap.cc:492] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7: Bootstrap starting.
I20251023 20:12:15.607991 14857 tablet_bootstrap.cc:492] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: Bootstrap starting.
I20251023 20:12:15.608330 14856 tablet_bootstrap.cc:654] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:15.608461 14858 tablet_bootstrap.cc:492] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: Bootstrap starting.
I20251023 20:12:15.608685 14857 tablet_bootstrap.cc:654] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:15.608718 14856 log.cc:826] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:15.609021 14857 log.cc:826] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:15.609143 14858 tablet_bootstrap.cc:654] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:15.609408 14858 log.cc:826] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:15.609549 14856 tablet_bootstrap.cc:492] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7: No bootstrap required, opened a new log
I20251023 20:12:15.609615 14856 ts_tablet_manager.cc:1403] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:12:15.609760 14857 tablet_bootstrap.cc:492] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: No bootstrap required, opened a new log
I20251023 20:12:15.609820 14857 ts_tablet_manager.cc:1403] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:12:15.610195 14858 tablet_bootstrap.cc:492] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: No bootstrap required, opened a new log
I20251023 20:12:15.610248 14858 ts_tablet_manager.cc:1403] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:12:15.611271 14857 raft_consensus.cc:359] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.611378 14857 raft_consensus.cc:385] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:15.611421 14857 raft_consensus.cc:740] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9108577656d1465bba696b56bcf833b2, State: Initialized, Role: FOLLOWER
I20251023 20:12:15.611517 14857 consensus_queue.cc:260] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.611804 14857 ts_tablet_manager.cc:1434] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:12:15.611933 14704 heartbeater.cc:499] Master 127.9.117.254:42395 was elected leader, sending a full tablet report...
I20251023 20:12:15.611835 14856 raft_consensus.cc:359] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.611990 14856 raft_consensus.cc:385] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:15.612027 14856 raft_consensus.cc:740] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2990814298403ab5d3de7e774f11c7, State: Initialized, Role: FOLLOWER
I20251023 20:12:15.612042 14858 raft_consensus.cc:359] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.612121 14856 consensus_queue.cc:260] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.612183 14858 raft_consensus.cc:385] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:15.612216 14858 raft_consensus.cc:740] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562f34df42634626abf96bf8e58a9912, State: Initialized, Role: FOLLOWER
I20251023 20:12:15.612314 14858 consensus_queue.cc:260] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.612385 14856 ts_tablet_manager.cc:1434] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251023 20:12:15.612495 14569 heartbeater.cc:499] Master 127.9.117.254:42395 was elected leader, sending a full tablet report...
I20251023 20:12:15.612562 14858 ts_tablet_manager.cc:1434] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:12:15.612715 14839 heartbeater.cc:499] Master 127.9.117.254:42395 was elected leader, sending a full tablet report...
W20251023 20:12:15.635860 14705 tablet.cc:2378] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:12:15.656975 14862 raft_consensus.cc:493] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:15.657126 14862 raft_consensus.cc:515] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.657477 14862 leader_election.cc:290] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562f34df42634626abf96bf8e58a9912 (127.9.117.195:36631), 3e2990814298403ab5d3de7e774f11c7 (127.9.117.193:34543)
I20251023 20:12:15.660245 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.657706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34543 (local address 127.9.117.194:35887)
1023 20:12:15.657784 (+ 78us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.657807 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.657878 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.658045 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.658046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.658048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:15.658311 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.658313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.659198 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.659199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:15.660005 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.660008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.660088 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.660093 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:12:15.660125 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:12:15.660164 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20251023 20:12:15.660673 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.657796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:35887 (local address 127.9.117.193:34543)
1023 20:12:15.657871 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.657874 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.657890 (+ 16us) server_negotiation.cc:408] Connection header received
1023 20:12:15.657957 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.657958 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.657970 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.658033 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:15.658416 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.659105 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.660105 (+ 1000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.660527 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.660537 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.660594 (+ 57us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.660624 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:12:15.661031 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.657706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:36631 (local address 127.9.117.194:57137)
1023 20:12:15.658127 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.658139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.658232 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.658431 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.658432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.658434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:15.658659 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.658661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.659501 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.659502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:15.660309 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:15.660312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.660874 (+ 562us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.660878 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:12:15.660939 (+ 61us) client_negotiation.cc:240] Negotiation successful
1023 20:12:15.660975 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":188,"thread_start_us":133,"threads_started":1}
I20251023 20:12:15.661091 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.657817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:57137 (local address 127.9.117.195:36631)
1023 20:12:15.657879 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.657882 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.658199 (+ 317us) server_negotiation.cc:408] Connection header received
1023 20:12:15.658315 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.658317 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.658328 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.658382 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:15.658748 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.659404 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.660413 (+ 1009us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.660829 (+ 416us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.660840 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.660973 (+ 133us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.661029 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:12:15.661036 14523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2420c6e108043e9825fd47b5c6380fe" candidate_uuid: "9108577656d1465bba696b56bcf833b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2990814298403ab5d3de7e774f11c7" is_pre_election: true
I20251023 20:12:15.661211 14523 raft_consensus.cc:2468] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9108577656d1465bba696b56bcf833b2 in term 0.
I20251023 20:12:15.661401 14793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2420c6e108043e9825fd47b5c6380fe" candidate_uuid: "9108577656d1465bba696b56bcf833b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562f34df42634626abf96bf8e58a9912" is_pre_election: true
I20251023 20:12:15.661473 14592 leader_election.cc:304] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2990814298403ab5d3de7e774f11c7, 9108577656d1465bba696b56bcf833b2; no voters:
I20251023 20:12:15.661556 14793 raft_consensus.cc:2468] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9108577656d1465bba696b56bcf833b2 in term 0.
I20251023 20:12:15.661620 14862 raft_consensus.cc:2804] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:15.661707 14862 raft_consensus.cc:493] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:15.661746 14862 raft_consensus.cc:3060] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:15.662432 14862 raft_consensus.cc:515] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.662601 14862 leader_election.cc:290] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [CANDIDATE]: Term 1 election: Requested vote from peers 562f34df42634626abf96bf8e58a9912 (127.9.117.195:36631), 3e2990814298403ab5d3de7e774f11c7 (127.9.117.193:34543)
I20251023 20:12:15.662830 14793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2420c6e108043e9825fd47b5c6380fe" candidate_uuid: "9108577656d1465bba696b56bcf833b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562f34df42634626abf96bf8e58a9912"
I20251023 20:12:15.662811 14523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2420c6e108043e9825fd47b5c6380fe" candidate_uuid: "9108577656d1465bba696b56bcf833b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2990814298403ab5d3de7e774f11c7"
I20251023 20:12:15.662918 14523 raft_consensus.cc:3060] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:15.662918 14793 raft_consensus.cc:3060] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:15.663915 14793 raft_consensus.cc:2468] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9108577656d1465bba696b56bcf833b2 in term 1.
I20251023 20:12:15.663916 14523 raft_consensus.cc:2468] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9108577656d1465bba696b56bcf833b2 in term 1.
I20251023 20:12:15.664160 14592 leader_election.cc:304] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2990814298403ab5d3de7e774f11c7, 9108577656d1465bba696b56bcf833b2; no voters:
I20251023 20:12:15.664263 14862 raft_consensus.cc:2804] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:15.664418 14862 raft_consensus.cc:697] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 1 LEADER]: Becoming Leader. State: Replica: 9108577656d1465bba696b56bcf833b2, State: Running, Role: LEADER
I20251023 20:12:15.664515 14862 consensus_queue.cc:237] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } }
I20251023 20:12:15.665403 14376 catalog_manager.cc:5649] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9108577656d1465bba696b56bcf833b2 (127.9.117.194). New cstate: current_term: 1 leader_uuid: "9108577656d1465bba696b56bcf833b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9108577656d1465bba696b56bcf833b2" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 45243 } health_report { overall_health: HEALTHY } } }
I20251023 20:12:15.677184 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.675131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.9.117.194:45243)
1023 20:12:15.675197 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.675199 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.675208 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:15.675249 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.675250 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.675267 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.675314 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:15.675565 (+ 251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.676047 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:15.676596 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:15.676796 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:15.676841 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:15.676940 (+ 99us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:15.677017 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.677078 (+ 61us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.677105 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251023 20:12:15.678869 14523 raft_consensus.cc:1275] T c2420c6e108043e9825fd47b5c6380fe P 3e2990814298403ab5d3de7e774f11c7 [term 1 FOLLOWER]: Refusing update from remote peer 9108577656d1465bba696b56bcf833b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:15.679167 14793 raft_consensus.cc:1275] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 1 FOLLOWER]: Refusing update from remote peer 9108577656d1465bba696b56bcf833b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:15.679206 14862 consensus_queue.cc:1048] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2990814298403ab5d3de7e774f11c7" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:15.679450 14862 consensus_queue.cc:1048] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562f34df42634626abf96bf8e58a9912" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 36631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:15.682467 14870 mvcc.cc:204] Tried to move back new op lower bound from 7214081374937886720 to 7214081374882480128. Current Snapshot: MvccSnapshot[applied={T|T < 7214081374937886720}]
I20251023 20:12:15.682592 14871 mvcc.cc:204] Tried to move back new op lower bound from 7214081374937886720 to 7214081374882480128. Current Snapshot: MvccSnapshot[applied={T|T < 7214081374937886720}]
I20251023 20:12:15.683925 14377 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34192:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:15.684028 14377 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34192:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:15.684779 14377 catalog_manager.cc:5936] T 00000000000000000000000000000000 P c1f33cd816204460aaf38142ad4c4615: Sending DeleteTablet for 3 replicas of tablet c2420c6e108043e9825fd47b5c6380fe
I20251023 20:12:15.685099 14773 tablet_service.cc:1552] Processing DeleteTablet for tablet c2420c6e108043e9825fd47b5c6380fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:15 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:57506
I20251023 20:12:15.685156 14638 tablet_service.cc:1552] Processing DeleteTablet for tablet c2420c6e108043e9825fd47b5c6380fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:15 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:43652
I20251023 20:12:15.685377 14878 tablet_replica.cc:333] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: stopping tablet replica
I20251023 20:12:15.685389 14879 tablet_replica.cc:333] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: stopping tablet replica
I20251023 20:12:15.685469 14879 raft_consensus.cc:2243] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:12:15.685489 14878 raft_consensus.cc:2243] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:15.685598 14878 raft_consensus.cc:2272] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:15.685637 14879 raft_consensus.cc:2272] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:15.685729 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 14441
I20251023 20:12:15.686990 14878 ts_tablet_manager.cc:1916] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:15.688438 14878 ts_tablet_manager.cc:1929] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:15.688503 14878 log.cc:1199] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/c2420c6e108043e9825fd47b5c6380fe
I20251023 20:12:15.688784 14878 ts_tablet_manager.cc:1950] T c2420c6e108043e9825fd47b5c6380fe P 562f34df42634626abf96bf8e58a9912: Deleting consensus metadata
I20251023 20:12:15.689162 14361 catalog_manager.cc:4985] TS 562f34df42634626abf96bf8e58a9912 (127.9.117.195:36631): tablet c2420c6e108043e9825fd47b5c6380fe (table test-table [id=38b0e399ae564f9f85a6ea9a7b4041c8]) successfully deleted
W20251023 20:12:15.690158 14362 connection.cc:537] client connection to 127.9.117.193:34543 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:12:15.690243 14362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:12:15.690619 14879 ts_tablet_manager.cc:1916] T c2420c6e108043e9825fd47b5c6380fe P 9108577656d1465bba696b56bcf833b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:15.690745 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.690609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34543 (local address 127.0.0.1:35594)
1023 20:12:15.690666 (+ 57us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.690707 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:34543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20251023 20:12:15.690889 14362 catalog_manager.cc:4712] TS 3e2990814298403ab5d3de7e774f11c7 (127.9.117.193:34543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2420c6e108043e9825fd47b5c6380fe: Network error: Client connection negotiation failed: client connection to 127.9.117.193:34543: connect: Connection refused (error 111)
I20251023 20:12:15.691640 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 14576
I20251023 20:12:15.696892 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 14711
I20251023 20:12:15.696978 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.696832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:45243 (local address 127.0.0.1:43676)
1023 20:12:15.696902 (+ 70us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.696936 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.194:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251023 20:12:15.701846 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 14346
2025-10-23T20:12:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2963 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-10-23T20:12:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:15Z Disabled control of system clock
I20251023 20:12:15.729285 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40905
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251023 20:12:15.811017 14884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:15.811224 14884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:15.811249 14884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:15.812829 14884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:15.812889 14884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:15.812903 14884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:15.812915 14884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:15.812928 14884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:15.814546 14884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:40905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:15.814817 14884 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:15.815040 14884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:15.818172 14890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.818228 14884 server_base.cc:1047] running on GCE node
W20251023 20:12:15.818166 14889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:15.818166 14892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.818676 14884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:15.818953 14884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:15.820176 14884 hybrid_clock.cc:648] HybridClock initialized: now 1761250335820157 us; error 34 us; skew 500 ppm
I20251023 20:12:15.821518 14884 webserver.cc:492] Webserver started at http://127.9.117.254:35779/ using document root <none> and password file <none>
I20251023 20:12:15.821733 14884 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:15.821774 14884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:15.821878 14884 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:15.822885 14884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "77cf8cae922145a98870693b9a615135"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "a53ce06ea233af3e2ae4aa57d3a8931b"
server_key_iv: "07f89de3437030294331af531d6462d7"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.823232 14884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "77cf8cae922145a98870693b9a615135"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "a53ce06ea233af3e2ae4aa57d3a8931b"
server_key_iv: "07f89de3437030294331af531d6462d7"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.824469 14884 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:15.825399 14898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.825731 14884 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:15.825831 14884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "77cf8cae922145a98870693b9a615135"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "a53ce06ea233af3e2ae4aa57d3a8931b"
server_key_iv: "07f89de3437030294331af531d6462d7"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.825919 14884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:15.830039 14884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:15.830354 14884 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:15.830479 14884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:15.835458 14884 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:40905
I20251023 20:12:15.835549 14950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:40905 every 8 connection(s)
I20251023 20:12:15.835899 14884 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:15.836549 14951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:15.839272 14951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135: Bootstrap starting.
I20251023 20:12:15.839871 14951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:15.840101 14951 log.cc:826] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:15.840906 14951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135: No bootstrap required, opened a new log
I20251023 20:12:15.842195 14951 raft_consensus.cc:359] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cf8cae922145a98870693b9a615135" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40905 } }
I20251023 20:12:15.842337 14951 raft_consensus.cc:385] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:15.842360 14951 raft_consensus.cc:740] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cf8cae922145a98870693b9a615135, State: Initialized, Role: FOLLOWER
I20251023 20:12:15.842471 14951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cf8cae922145a98870693b9a615135" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40905 } }
I20251023 20:12:15.842546 14951 raft_consensus.cc:399] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:15.842590 14951 raft_consensus.cc:493] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:15.842621 14951 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:15.843190 14951 raft_consensus.cc:515] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cf8cae922145a98870693b9a615135" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40905 } }
I20251023 20:12:15.843290 14951 leader_election.cc:304] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77cf8cae922145a98870693b9a615135; no voters:
I20251023 20:12:15.843443 14951 leader_election.cc:290] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:15.843528 14954 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:15.843550 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 14884
I20251023 20:12:15.843654 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:15.843729 14951 sys_catalog.cc:565] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:15.843791 14954 raft_consensus.cc:697] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [term 1 LEADER]: Becoming Leader. State: Replica: 77cf8cae922145a98870693b9a615135, State: Running, Role: LEADER
I20251023 20:12:15.843873 9687 external_mini_cluster.cc:1468] Setting key 8f16ca448819851400ce807df982b931
I20251023 20:12:15.843902 14954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cf8cae922145a98870693b9a615135" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40905 } }
I20251023 20:12:15.844305 14956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77cf8cae922145a98870693b9a615135. Latest consensus state: current_term: 1 leader_uuid: "77cf8cae922145a98870693b9a615135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cf8cae922145a98870693b9a615135" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40905 } } }
I20251023 20:12:15.844401 14956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:15.844282 14955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77cf8cae922145a98870693b9a615135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cf8cae922145a98870693b9a615135" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40905 } } }
I20251023 20:12:15.844704 14955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:15.844991 14962 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:15.845503 14962 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:15.846961 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.844526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.9.117.254:40905)
1023 20:12:15.844779 (+ 253us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.844789 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.845050 (+ 261us) server_negotiation.cc:408] Connection header received
1023 20:12:15.845180 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.845186 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.845237 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.845369 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:15.846470 (+ 1101us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:15.846474 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:15.846475 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:15.846481 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:15.846521 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:15.846530 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:15.846635 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:15.846721 (+ 86us) server_negotiation.cc:300] Negotiation successful
1023 20:12:15.846776 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20251023 20:12:15.847438 14962 catalog_manager.cc:1357] Generated new cluster ID: d07ea04e5e0b4bffb1739dd27fef10b5
I20251023 20:12:15.847493 14962 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:15.872618 14962 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:15.872884 14962 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:15.881903 14962 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135: Generated new TSK 0
I20251023 20:12:15.882119 14962 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:12:15.888646 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40905
--builtin_ntp_servers=127.9.117.212:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251023 20:12:15.973191 14975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:15.973362 14975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:15.973387 14975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:15.974893 14975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:15.974943 14975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:15.974970 14975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:15.976434 14975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:15.976678 14975 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:15.976948 14975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:15.979576 14981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:15.979607 14983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:15.979650 14980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:15.979852 14975 server_base.cc:1047] running on GCE node
I20251023 20:12:15.980101 14975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:15.980311 14975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:15.981451 14975 hybrid_clock.cc:648] HybridClock initialized: now 1761250335981436 us; error 34 us; skew 500 ppm
I20251023 20:12:15.982692 14975 webserver.cc:492] Webserver started at http://127.9.117.193:33865/ using document root <none> and password file <none>
I20251023 20:12:15.982920 14975 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:15.982967 14975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:15.983089 14975 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:15.984093 14975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "109a88e85a7e4042a865325c21020186"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "1e579990149acf97857f18ee6d056b9f"
server_key_iv: "e311af5038d3ae8635e32bed6ca48df8"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.984444 14975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "109a88e85a7e4042a865325c21020186"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "1e579990149acf97857f18ee6d056b9f"
server_key_iv: "e311af5038d3ae8635e32bed6ca48df8"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.985767 14975 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:12:15.986482 14989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.986670 14975 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:15.986758 14975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "109a88e85a7e4042a865325c21020186"
format_stamp: "Formatted at 2025-10-23 20:12:15 on dist-test-slave-x62p"
server_key: "1e579990149acf97857f18ee6d056b9f"
server_key_iv: "e311af5038d3ae8635e32bed6ca48df8"
server_key_version: "encryptionkey@0"
I20251023 20:12:15.986825 14975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:15.990445 14975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:15.990737 14975 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:15.990868 14975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:15.991067 14975 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:15.991360 14975 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:15.991392 14975 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.991428 14975 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:15.991459 14975 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:15.997604 14975 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:46205
I20251023 20:12:15.997687 15102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:46205 every 8 connection(s)
I20251023 20:12:15.998001 14975 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:16.000852 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.998792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:44339 (local address 127.9.117.254:40905)
1023 20:12:15.998862 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:15.998864 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:15.999081 (+ 217us) server_negotiation.cc:408] Connection header received
1023 20:12:15.999634 (+ 553us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:15.999635 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:15.999642 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:15.999707 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.000519 (+ 812us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.000520 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.000520 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.000527 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.000544 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.000548 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.000602 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.000707 (+ 105us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.000760 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:12:16.001071 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:15.998749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40905 (local address 127.9.117.193:44339)
1023 20:12:15.998964 (+ 215us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:15.998993 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:15.999528 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:15.999773 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:15.999777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:15.999786 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:16.000396 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:12:16.000396 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:16.000416 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:12:16.000436 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:16.000629 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:16.000633 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:16.000636 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:16.000761 (+ 125us) client_negotiation.cc:240] Negotiation successful
1023 20:12:16.000823 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20251023 20:12:16.001582 15103 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40905
I20251023 20:12:16.001724 15103 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:16.001897 15103 heartbeater.cc:507] Master 127.9.117.254:40905 requested a full tablet report, sending...
I20251023 20:12:16.002384 14915 ts_manager.cc:194] Registered new tserver with Master: 109a88e85a7e4042a865325c21020186 (127.9.117.193:46205)
I20251023 20:12:16.003374 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 14975
I20251023 20:12:16.003463 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:16.003638 9687 external_mini_cluster.cc:1468] Setting key 347db3ba3eb0e5bdaf5532c4472f41b5
I20251023 20:12:16.004624 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40905
--builtin_ntp_servers=127.9.117.212:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251023 20:12:16.094213 15106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:16.094426 15106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:16.094451 15106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:16.095909 15106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:16.095956 15106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:16.095983 15106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:16.097534 15106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:16.097782 15106 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:16.098011 15106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:16.100854 15114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:16.100952 15106 server_base.cc:1047] running on GCE node
W20251023 20:12:16.100884 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:16.100903 15112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:16.101249 15106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:16.101470 15106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:16.102619 15106 hybrid_clock.cc:648] HybridClock initialized: now 1761250336102606 us; error 34 us; skew 500 ppm
I20251023 20:12:16.103904 15106 webserver.cc:492] Webserver started at http://127.9.117.194:34879/ using document root <none> and password file <none>
I20251023 20:12:16.104115 15106 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:16.104163 15106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:16.104295 15106 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:16.105324 15106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "474233ce15e642f4b1a4cc411562af17"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "3da57d981614039d5908e2d8f1ac032e"
server_key_iv: "e1d288870ba0ff40b40f32a35e15e66c"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.105643 15106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "474233ce15e642f4b1a4cc411562af17"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "3da57d981614039d5908e2d8f1ac032e"
server_key_iv: "e1d288870ba0ff40b40f32a35e15e66c"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.106992 15106 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:16.107962 15120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:16.108129 15106 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:16.108203 15106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "474233ce15e642f4b1a4cc411562af17"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "3da57d981614039d5908e2d8f1ac032e"
server_key_iv: "e1d288870ba0ff40b40f32a35e15e66c"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.108273 15106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:16.111976 15106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:16.112252 15106 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:16.112366 15106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:16.112564 15106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:16.112922 15106 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:16.112957 15106 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:16.112993 15106 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:16.113018 15106 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:16.119195 15106 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:33955
I20251023 20:12:16.119283 15233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:33955 every 8 connection(s)
I20251023 20:12:16.119588 15106 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:12:16.119877 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 15106
I20251023 20:12:16.119973 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:16.120167 9687 external_mini_cluster.cc:1468] Setting key 178f57b23c3e29b77322c8f2db862904
I20251023 20:12:16.121331 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40905
--builtin_ntp_servers=127.9.117.212:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251023 20:12:16.122534 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.120670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:38003 (local address 127.9.117.254:40905)
1023 20:12:16.120717 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.120719 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.120753 (+ 34us) server_negotiation.cc:408] Connection header received
1023 20:12:16.121320 (+ 567us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.121321 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.121330 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.121398 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.122251 (+ 853us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.122252 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.122252 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.122258 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.122275 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.122280 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.122345 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.122412 (+ 67us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.122431 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:12:16.122723 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.120432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40905 (local address 127.9.117.194:38003)
1023 20:12:16.120634 (+ 202us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.120662 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:16.121221 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:16.121495 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:16.121501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:16.121515 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:16.122120 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:12:16.122120 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:16.122143 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:12:16.122164 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:16.122347 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:16.122349 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:16.122351 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:12:16.122427 (+ 76us) client_negotiation.cc:240] Negotiation successful
1023 20:12:16.122481 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20251023 20:12:16.123320 15234 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40905
I20251023 20:12:16.123432 15234 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:16.123579 15234 heartbeater.cc:507] Master 127.9.117.254:40905 requested a full tablet report, sending...
I20251023 20:12:16.123914 14915 ts_manager.cc:194] Registered new tserver with Master: 474233ce15e642f4b1a4cc411562af17 (127.9.117.194:33955)
W20251023 20:12:16.209908 15237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:16.210093 15237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:16.210114 15237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:16.211696 15237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:16.211750 15237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:16.211776 15237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:16.213696 15237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:16.213984 15237 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:16.214226 15237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:16.216954 15242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:16.216964 15245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:16.216965 15243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:16.217149 15237 server_base.cc:1047] running on GCE node
I20251023 20:12:16.217377 15237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:16.217597 15237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:16.218750 15237 hybrid_clock.cc:648] HybridClock initialized: now 1761250336218728 us; error 38 us; skew 500 ppm
I20251023 20:12:16.219978 15237 webserver.cc:492] Webserver started at http://127.9.117.195:36369/ using document root <none> and password file <none>
I20251023 20:12:16.220194 15237 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:16.220247 15237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:16.220369 15237 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:16.221434 15237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "1e8132a370874afe8a8da72e283676a2"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "c7cc3fa6edf36c2b882ae80e32fe70c4"
server_key_iv: "bf2ca80e9e4daca127a1900683e4b2b4"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.221771 15237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "1e8132a370874afe8a8da72e283676a2"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "c7cc3fa6edf36c2b882ae80e32fe70c4"
server_key_iv: "bf2ca80e9e4daca127a1900683e4b2b4"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.223842 15237 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251023 20:12:16.224869 15251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:16.225095 15237 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:16.225178 15237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "1e8132a370874afe8a8da72e283676a2"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "c7cc3fa6edf36c2b882ae80e32fe70c4"
server_key_iv: "bf2ca80e9e4daca127a1900683e4b2b4"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.225250 15237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:16.229072 15237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:16.229373 15237 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:16.229487 15237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:16.229681 15237 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:16.229981 15237 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:16.230015 15237 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:16.230051 15237 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:16.230082 15237 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:16.236255 15237 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:44859
I20251023 20:12:16.236331 15364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:44859 every 8 connection(s)
I20251023 20:12:16.236640 15237 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:12:16.239293 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.237395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:56901 (local address 127.9.117.254:40905)
1023 20:12:16.237451 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.237456 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.237662 (+ 206us) server_negotiation.cc:408] Connection header received
1023 20:12:16.238186 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.238187 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.238193 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.238247 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.239016 (+ 769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.239018 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.239019 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.239024 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.239047 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.239051 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.239107 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.239192 (+ 85us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.239214 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:12:16.239483 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.237343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40905 (local address 127.9.117.195:56901)
1023 20:12:16.237557 (+ 214us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.237583 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:16.238089 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:16.238310 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:16.238314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:16.238323 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:16.238863 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:12:16.238863 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:16.238885 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:12:16.238907 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:16.239110 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:16.239114 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:16.239117 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:16.239217 (+ 100us) client_negotiation.cc:240] Negotiation successful
1023 20:12:16.239270 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20251023 20:12:16.240023 15365 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40905
I20251023 20:12:16.240108 15365 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:16.240242 15365 heartbeater.cc:507] Master 127.9.117.254:40905 requested a full tablet report, sending...
I20251023 20:12:16.240597 14915 ts_manager.cc:194] Registered new tserver with Master: 1e8132a370874afe8a8da72e283676a2 (127.9.117.195:44859)
I20251023 20:12:16.246433 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 15237
I20251023 20:12:16.246522 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:16.246668 9687 external_mini_cluster.cc:1468] Setting key ede6158cc7d94601a200c22418d45aee
I20251023 20:12:16.247661 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:12:16.250187 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.249019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.9.117.254:40905)
1023 20:12:16.249082 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.249084 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.249178 (+ 94us) server_negotiation.cc:408] Connection header received
1023 20:12:16.249223 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.249224 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.249231 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.249284 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.249981 (+ 697us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.249983 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.249983 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.249988 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.250004 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.250008 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.250060 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.250114 (+ 54us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.250130 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251023 20:12:16.251855 14915 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:12:16.253207 14915 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:16.257556 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.255442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:46205 (local address 127.0.0.1:42856)
1023 20:12:16.255615 (+ 173us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.255645 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:16.255710 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:16.256307 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:16.256308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:16.256310 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:16.256899 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:12:16.256899 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:16.256924 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:12:16.256944 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:16.257372 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:16.257378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:16.257381 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:16.257449 (+ 68us) client_negotiation.cc:240] Negotiation successful
1023 20:12:16.257466 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20251023 20:12:16.257571 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.255589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.9.117.193:46205)
1023 20:12:16.256159 (+ 570us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.256167 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.256177 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:16.256201 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.256203 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.256210 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.256433 (+ 223us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.257247 (+ 814us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.257250 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.257250 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.257253 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.257280 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.257285 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.257379 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.257462 (+ 83us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.257477 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":114,"threads_started":1}
I20251023 20:12:16.257560 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.255448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:44859 (local address 127.0.0.1:46988)
1023 20:12:16.255995 (+ 547us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.256001 (+ 6us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:16.256035 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:16.256462 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:16.256463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:16.256466 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:16.257123 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:12:16.257124 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:16.257128 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:12:16.257136 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:16.257403 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:16.257404 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:16.257404 (+ 0us) client_negotiation.cc:769] Sending connection context
1023 20:12:16.257460 (+ 56us) client_negotiation.cc:240] Negotiation successful
1023 20:12:16.257477 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":124,"threads_started":1}
I20251023 20:12:16.257822 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.255707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.9.117.195:44859)
1023 20:12:16.255973 (+ 266us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.255979 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.256056 (+ 77us) server_negotiation.cc:408] Connection header received
1023 20:12:16.256082 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.256084 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.256091 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.256151 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.257226 (+ 1075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.257234 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.257234 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.257240 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.257294 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.257301 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.257423 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.257470 (+ 47us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.257485 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":89,"threads_started":1}
I20251023 20:12:16.258282 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.255446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:33955 (local address 127.0.0.1:55784)
1023 20:12:16.255872 (+ 426us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.255881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:16.255936 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:16.256153 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:16.256159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:16.256167 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:16.257924 (+ 1757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:12:16.257925 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:16.257931 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:12:16.257944 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:16.258146 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:16.258147 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:16.258148 (+ 1us) client_negotiation.cc:769] Sending connection context
1023 20:12:16.258202 (+ 54us) client_negotiation.cc:240] Negotiation successful
1023 20:12:16.258219 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"spinlock_wait_cycles":768,"thread_start_us":121,"threads_started":1}
I20251023 20:12:16.258435 15299 tablet_service.cc:1505] Processing CreateTablet for tablet f27e8768565c4d2e9a9a773f239e9288 (DEFAULT_TABLE table=test-table [id=f7fed70c36fa40f19813ad696bcae1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:16.258539 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.255708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.9.117.194:33955)
1023 20:12:16.255888 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.255895 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.255951 (+ 56us) server_negotiation.cc:408] Connection header received
1023 20:12:16.255983 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.255986 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.255995 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.256074 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.258006 (+ 1932us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.258015 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.258015 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.258020 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.258072 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.258079 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.258152 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.258208 (+ 56us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.258223 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20251023 20:12:16.258756 15299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e8768565c4d2e9a9a773f239e9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:16.259572 15168 tablet_service.cc:1505] Processing CreateTablet for tablet f27e8768565c4d2e9a9a773f239e9288 (DEFAULT_TABLE table=test-table [id=f7fed70c36fa40f19813ad696bcae1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:16.259816 15168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e8768565c4d2e9a9a773f239e9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:16.259713 15037 tablet_service.cc:1505] Processing CreateTablet for tablet f27e8768565c4d2e9a9a773f239e9288 (DEFAULT_TABLE table=test-table [id=f7fed70c36fa40f19813ad696bcae1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:16.260015 15037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e8768565c4d2e9a9a773f239e9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:16.261780 15379 tablet_bootstrap.cc:492] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Bootstrap starting.
I20251023 20:12:16.262416 15379 tablet_bootstrap.cc:654] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:16.262634 15381 tablet_bootstrap.cc:492] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186: Bootstrap starting.
I20251023 20:12:16.262673 15379 log.cc:826] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:16.263288 15381 tablet_bootstrap.cc:654] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:16.263280 15380 tablet_bootstrap.cc:492] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Bootstrap starting.
I20251023 20:12:16.263375 15379 tablet_bootstrap.cc:492] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: No bootstrap required, opened a new log
I20251023 20:12:16.263454 15379 ts_tablet_manager.cc:1403] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:12:16.263607 15381 log.cc:826] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:16.263927 15380 tablet_bootstrap.cc:654] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:16.264215 15380 log.cc:826] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:16.264393 15381 tablet_bootstrap.cc:492] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186: No bootstrap required, opened a new log
I20251023 20:12:16.264503 15381 ts_tablet_manager.cc:1403] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:12:16.265022 15380 tablet_bootstrap.cc:492] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: No bootstrap required, opened a new log
I20251023 20:12:16.265043 15379 raft_consensus.cc:359] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.265137 15380 ts_tablet_manager.cc:1403] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:12:16.265165 15379 raft_consensus.cc:385] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:16.265199 15379 raft_consensus.cc:740] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8132a370874afe8a8da72e283676a2, State: Initialized, Role: FOLLOWER
I20251023 20:12:16.265340 15379 consensus_queue.cc:260] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.265595 15379 ts_tablet_manager.cc:1434] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:12:16.265709 15365 heartbeater.cc:499] Master 127.9.117.254:40905 was elected leader, sending a full tablet report...
I20251023 20:12:16.266158 15381 raft_consensus.cc:359] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.266305 15381 raft_consensus.cc:385] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:16.266338 15381 raft_consensus.cc:740] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109a88e85a7e4042a865325c21020186, State: Initialized, Role: FOLLOWER
I20251023 20:12:16.266460 15381 consensus_queue.cc:260] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.266732 15381 ts_tablet_manager.cc:1434] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:12:16.266778 15103 heartbeater.cc:499] Master 127.9.117.254:40905 was elected leader, sending a full tablet report...
I20251023 20:12:16.266774 15380 raft_consensus.cc:359] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.266902 15380 raft_consensus.cc:385] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:16.266939 15380 raft_consensus.cc:740] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474233ce15e642f4b1a4cc411562af17, State: Initialized, Role: FOLLOWER
I20251023 20:12:16.267051 15380 consensus_queue.cc:260] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.267282 15380 ts_tablet_manager.cc:1434] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:12:16.267311 15234 heartbeater.cc:499] Master 127.9.117.254:40905 was elected leader, sending a full tablet report...
W20251023 20:12:16.370201 15235 tablet.cc:2378] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:12:16.441051 15386 raft_consensus.cc:493] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:16.441183 15386 raft_consensus.cc:515] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.441560 15386 leader_election.cc:290] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474233ce15e642f4b1a4cc411562af17 (127.9.117.194:33955), 1e8132a370874afe8a8da72e283676a2 (127.9.117.195:44859)
I20251023 20:12:16.443019 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.441731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:33955 (local address 127.9.117.193:41199)
1023 20:12:16.441807 (+ 76us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.441840 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:16.441892 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:16.442067 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:16.442067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:16.442070 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:16.442601 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:12:16.442602 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:16.442611 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:12:16.442628 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:16.442899 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:16.442900 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:16.442901 (+ 1us) client_negotiation.cc:769] Sending connection context
1023 20:12:16.442945 (+ 44us) client_negotiation.cc:240] Negotiation successful
1023 20:12:16.442958 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20251023 20:12:16.443090 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.441823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:41199 (local address 127.9.117.194:33955)
1023 20:12:16.441893 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.441895 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.441906 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:12:16.441965 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.441966 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.441975 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.442051 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.442769 (+ 718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.442772 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.442772 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.442779 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.442801 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.442805 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.442897 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.442953 (+ 56us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.442972 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251023 20:12:16.443431 15188 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f27e8768565c4d2e9a9a773f239e9288" candidate_uuid: "109a88e85a7e4042a865325c21020186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474233ce15e642f4b1a4cc411562af17" is_pre_election: true
I20251023 20:12:16.443653 15188 raft_consensus.cc:2468] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 109a88e85a7e4042a865325c21020186 in term 0.
I20251023 20:12:16.443758 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.441809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:55779 (local address 127.9.117.195:44859)
1023 20:12:16.441866 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.441868 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.442060 (+ 192us) server_negotiation.cc:408] Connection header received
1023 20:12:16.442095 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.442096 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.442104 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.442162 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.443443 (+ 1281us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.443444 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.443445 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.443450 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.443471 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.443476 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.443556 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.443670 (+ 114us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.443689 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:12:16.443795 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.441731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:44859 (local address 127.9.117.193:55779)
1023 20:12:16.441974 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.441985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:16.442043 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:16.442180 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:16.442180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:16.442182 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:16.443347 (+ 1165us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:12:16.443348 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:16.443354 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:12:16.443365 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:16.443579 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:16.443580 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:16.443581 (+ 1us) client_negotiation.cc:769] Sending connection context
1023 20:12:16.443639 (+ 58us) client_negotiation.cc:240] Negotiation successful
1023 20:12:16.443659 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":16,"thread_start_us":146,"threads_started":1}
I20251023 20:12:16.443908 14991 leader_election.cc:304] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 109a88e85a7e4042a865325c21020186, 474233ce15e642f4b1a4cc411562af17; no voters:
I20251023 20:12:16.444041 15386 raft_consensus.cc:2804] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:16.444087 15386 raft_consensus.cc:493] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:16.444118 15386 raft_consensus.cc:3060] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:16.444092 15319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f27e8768565c4d2e9a9a773f239e9288" candidate_uuid: "109a88e85a7e4042a865325c21020186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8132a370874afe8a8da72e283676a2" is_pre_election: true
I20251023 20:12:16.444252 15319 raft_consensus.cc:2468] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 109a88e85a7e4042a865325c21020186 in term 0.
I20251023 20:12:16.445012 15386 raft_consensus.cc:515] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.445197 15386 leader_election.cc:290] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [CANDIDATE]: Term 1 election: Requested vote from peers 474233ce15e642f4b1a4cc411562af17 (127.9.117.194:33955), 1e8132a370874afe8a8da72e283676a2 (127.9.117.195:44859)
I20251023 20:12:16.445372 15319 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f27e8768565c4d2e9a9a773f239e9288" candidate_uuid: "109a88e85a7e4042a865325c21020186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8132a370874afe8a8da72e283676a2"
I20251023 20:12:16.445446 15319 raft_consensus.cc:3060] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:16.446480 15319 raft_consensus.cc:2468] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 109a88e85a7e4042a865325c21020186 in term 1.
I20251023 20:12:16.446617 15188 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f27e8768565c4d2e9a9a773f239e9288" candidate_uuid: "109a88e85a7e4042a865325c21020186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474233ce15e642f4b1a4cc411562af17"
I20251023 20:12:16.446703 15188 raft_consensus.cc:3060] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:16.446708 14990 leader_election.cc:304] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 109a88e85a7e4042a865325c21020186, 1e8132a370874afe8a8da72e283676a2; no voters:
I20251023 20:12:16.446801 15386 raft_consensus.cc:2804] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:16.446955 15386 raft_consensus.cc:697] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [term 1 LEADER]: Becoming Leader. State: Replica: 109a88e85a7e4042a865325c21020186, State: Running, Role: LEADER
I20251023 20:12:16.447068 15386 consensus_queue.cc:237] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } }
I20251023 20:12:16.447860 14915 catalog_manager.cc:5649] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 109a88e85a7e4042a865325c21020186 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "109a88e85a7e4042a865325c21020186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "109a88e85a7e4042a865325c21020186" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 } health_report { overall_health: UNKNOWN } } }
I20251023 20:12:16.447990 15188 raft_consensus.cc:2468] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 109a88e85a7e4042a865325c21020186 in term 1.
I20251023 20:12:16.487135 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.486050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.9.117.193:46205)
1023 20:12:16.486115 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.486118 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.486135 (+ 17us) server_negotiation.cc:408] Connection header received
1023 20:12:16.486188 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.486189 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.486203 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.486253 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.486888 (+ 635us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.486890 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.486890 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.486896 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.486912 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.486915 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.486966 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.487025 (+ 59us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.487042 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20251023 20:12:16.487221 15366 tablet.cc:2378] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:12:16.488802 15319 raft_consensus.cc:1275] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 1 FOLLOWER]: Refusing update from remote peer 109a88e85a7e4042a865325c21020186: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:16.488811 15188 raft_consensus.cc:1275] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 1 FOLLOWER]: Refusing update from remote peer 109a88e85a7e4042a865325c21020186: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:16.489121 15386 consensus_queue.cc:1048] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8132a370874afe8a8da72e283676a2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:16.489257 15389 consensus_queue.cc:1048] T f27e8768565c4d2e9a9a773f239e9288 P 109a88e85a7e4042a865325c21020186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474233ce15e642f4b1a4cc411562af17" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:16.495128 15392 mvcc.cc:204] Tried to move back new op lower bound from 7214081378255360000 to 7214081378087755776. Current Snapshot: MvccSnapshot[applied={T|T < 7214081378255360000}]
I20251023 20:12:16.499075 14914 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45664:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:16.499269 14914 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45664:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:16.501751 14914 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 77cf8cae922145a98870693b9a615135: Sending DeleteTablet for 3 replicas of tablet f27e8768565c4d2e9a9a773f239e9288
I20251023 20:12:16.502290 15168 tablet_service.cc:1552] Processing DeleteTablet for tablet f27e8768565c4d2e9a9a773f239e9288 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:16 UTC) from {username='slave'} at 127.0.0.1:55784
I20251023 20:12:16.502312 15167 tablet_service.cc:1552] Processing DeleteTablet for tablet f27e8768565c4d2e9a9a773f239e9288 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:16 UTC) from {username='slave'} at 127.0.0.1:55784
I20251023 20:12:16.502719 15037 tablet_service.cc:1552] Processing DeleteTablet for tablet f27e8768565c4d2e9a9a773f239e9288 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:16 UTC) from {username='slave'} at 127.0.0.1:42856
I20251023 20:12:16.502724 15299 tablet_service.cc:1552] Processing DeleteTablet for tablet f27e8768565c4d2e9a9a773f239e9288 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:16 UTC) from {username='slave'} at 127.0.0.1:46988
I20251023 20:12:16.503154 15402 tablet_replica.cc:333] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: stopping tablet replica
I20251023 20:12:16.503377 15402 raft_consensus.cc:2243] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:16.503530 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 14975
I20251023 20:12:16.504707 15402 raft_consensus.cc:2272] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:16.505117 15404 tablet_replica.cc:333] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: stopping tablet replica
I20251023 20:12:16.505293 15404 raft_consensus.cc:2243] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:16.505472 15404 raft_consensus.cc:2272] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:16.505752 15402 ts_tablet_manager.cc:1916] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:16.508908 15402 ts_tablet_manager.cc:1929] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:16.509018 15402 log.cc:1199] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/f27e8768565c4d2e9a9a773f239e9288
I20251023 20:12:16.509351 15402 ts_tablet_manager.cc:1950] T f27e8768565c4d2e9a9a773f239e9288 P 474233ce15e642f4b1a4cc411562af17: Deleting consensus metadata
I20251023 20:12:16.509361 15404 ts_tablet_manager.cc:1916] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:16.509931 14900 catalog_manager.cc:4985] TS 474233ce15e642f4b1a4cc411562af17 (127.9.117.194:33955): tablet f27e8768565c4d2e9a9a773f239e9288 (table test-table [id=f7fed70c36fa40f19813ad696bcae1f6]) successfully deleted
W20251023 20:12:16.510113 14900 catalog_manager.cc:4952] TS 474233ce15e642f4b1a4cc411562af17 (127.9.117.194:33955): delete failed for tablet f27e8768565c4d2e9a9a773f239e9288 because the tablet was not found. No further retry: Not found: Tablet not found: f27e8768565c4d2e9a9a773f239e9288
I20251023 20:12:16.510967 15404 ts_tablet_manager.cc:1929] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:16.511044 15404 log.cc:1199] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/f27e8768565c4d2e9a9a773f239e9288
I20251023 20:12:16.511279 15404 ts_tablet_manager.cc:1950] T f27e8768565c4d2e9a9a773f239e9288 P 1e8132a370874afe8a8da72e283676a2: Deleting consensus metadata
I20251023 20:12:16.511711 14899 catalog_manager.cc:4985] TS 1e8132a370874afe8a8da72e283676a2 (127.9.117.195:44859): tablet f27e8768565c4d2e9a9a773f239e9288 (table test-table [id=f7fed70c36fa40f19813ad696bcae1f6]) successfully deleted
W20251023 20:12:16.512614 14902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:46205 (error 108)
I20251023 20:12:16.513152 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.513016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:46205 (local address 127.0.0.1:42886)
1023 20:12:16.513077 (+ 61us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.513117 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:46205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20251023 20:12:16.513263 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 15106
W20251023 20:12:16.513278 14902 catalog_manager.cc:4712] TS 109a88e85a7e4042a865325c21020186 (127.9.117.193:46205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f27e8768565c4d2e9a9a773f239e9288: Network error: Client connection negotiation failed: client connection to 127.9.117.193:46205: connect: Connection refused (error 111)
I20251023 20:12:16.518635 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 15237
I20251023 20:12:16.523530 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 14884
2025-10-23T20:12:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (821 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251023 20:12:16.544131 9687 mini_oidc.cc:150] Starting JWKS server
I20251023 20:12:16.545351 9687 webserver.cc:492] Webserver started at https://127.0.0.1:37697/ using document root <none> and password file <none>
I20251023 20:12:16.545430 9687 mini_oidc.cc:186] Starting OIDC Discovery server
I20251023 20:12:16.545555 9687 webserver.cc:492] Webserver started at http://127.0.0.1:40937/ using document root <none> and password file <none>
2025-10-23T20:12:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:16Z Disabled control of system clock
I20251023 20:12:16.553176 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:34609
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:34609
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/testchainca.cert with env {}
W20251023 20:12:16.635264 15411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:16.635461 15411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:16.635484 15411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:16.637845 15411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:16.637903 15411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:16.637916 15411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:16.637929 15411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:16.637941 15411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251023 20:12:16.637987 15411 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251023 20:12:16.638002 15411 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37697/jwks.json
I20251023 20:12:16.641031 15411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:34609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:34609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:16.641386 15411 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:16.641626 15411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:16.644421 15419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:16.644423 15417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:16.644699 15411 server_base.cc:1047] running on GCE node
W20251023 20:12:16.644424 15416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:16.645038 15411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:16.645303 15411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:16.646440 15411 hybrid_clock.cc:648] HybridClock initialized: now 1761250336646431 us; error 30 us; skew 500 ppm
I20251023 20:12:16.647668 15411 webserver.cc:492] Webserver started at http://127.9.117.254:45273/ using document root <none> and password file <none>
I20251023 20:12:16.647897 15411 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:16.647941 15411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:16.648062 15411 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:16.649164 15411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "7d97cdcb51184aebbb7288391f19ceaf"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "c8b448fe0e162763882d9aee51327857"
server_key_iv: "150b651ff62dccad049637541fc7f391"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.649493 15411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "7d97cdcb51184aebbb7288391f19ceaf"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "c8b448fe0e162763882d9aee51327857"
server_key_iv: "150b651ff62dccad049637541fc7f391"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.650795 15411 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:16.651628 15425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:16.651818 15411 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:16.651903 15411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "7d97cdcb51184aebbb7288391f19ceaf"
format_stamp: "Formatted at 2025-10-23 20:12:16 on dist-test-slave-x62p"
server_key: "c8b448fe0e162763882d9aee51327857"
server_key_iv: "150b651ff62dccad049637541fc7f391"
server_key_version: "encryptionkey@0"
I20251023 20:12:16.651983 15411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:16.666971 15411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:16.667328 15411 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:16.667475 15411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:16.671828 15411 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:34609
I20251023 20:12:16.671875 15477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:34609 every 8 connection(s)
I20251023 20:12:16.672223 15411 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:16.672853 15478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:16.675372 15478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf: Bootstrap starting.
I20251023 20:12:16.675971 15478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:16.676210 15478 log.cc:826] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:16.677052 15478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf: No bootstrap required, opened a new log
I20251023 20:12:16.677564 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 15411
I20251023 20:12:16.677651 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:16.677811 9687 external_mini_cluster.cc:1468] Setting key e29e62d4243c0d49a207b0c47b18527d
I20251023 20:12:16.678737 15478 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d97cdcb51184aebbb7288391f19ceaf" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34609 } }
I20251023 20:12:16.678901 15478 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:16.678938 15478 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d97cdcb51184aebbb7288391f19ceaf, State: Initialized, Role: FOLLOWER
I20251023 20:12:16.679076 15478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d97cdcb51184aebbb7288391f19ceaf" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34609 } }
I20251023 20:12:16.679160 15478 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:16.679190 15478 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:16.679240 15478 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:16.679946 15478 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d97cdcb51184aebbb7288391f19ceaf" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34609 } }
I20251023 20:12:16.680066 15478 leader_election.cc:304] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d97cdcb51184aebbb7288391f19ceaf; no voters:
I20251023 20:12:16.680204 15478 leader_election.cc:290] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:16.680270 15483 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:16.680416 15478 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:16.680439 15483 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [term 1 LEADER]: Becoming Leader. State: Replica: 7d97cdcb51184aebbb7288391f19ceaf, State: Running, Role: LEADER
I20251023 20:12:16.680529 15483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d97cdcb51184aebbb7288391f19ceaf" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34609 } }
I20251023 20:12:16.680928 15483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d97cdcb51184aebbb7288391f19ceaf. Latest consensus state: current_term: 1 leader_uuid: "7d97cdcb51184aebbb7288391f19ceaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d97cdcb51184aebbb7288391f19ceaf" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34609 } } }
I20251023 20:12:16.681038 15483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:16.680945 15484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d97cdcb51184aebbb7288391f19ceaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d97cdcb51184aebbb7288391f19ceaf" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34609 } } }
I20251023 20:12:16.681255 15484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf [sys.catalog]: This master's current role is: LEADER
W20251023 20:12:16.683135 15499 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:12:16.683207 15499 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:12:16.683274 15490 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:16.683516 15490 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:16.683584 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.678345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.9.117.254:34609)
1023 20:12:16.678610 (+ 265us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.678620 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.678643 (+ 23us) server_negotiation.cc:408] Connection header received
1023 20:12:16.678759 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.678765 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.678836 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.678982 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:16.679414 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:16.680675 (+ 1261us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:16.681569 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:16.681851 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:16.682451 (+ 600us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:16.682458 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:16.682458 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:16.683009 (+ 551us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:16.683049 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:16.683059 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:16.683142 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.683364 (+ 222us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.683423 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20251023 20:12:16.684592 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:12:16.685205 15490 catalog_manager.cc:1357] Generated new cluster ID: 6c968dbb8e9e4c6980b61694eaa9cbbc
I20251023 20:12:16.685261 15490 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:16.695895 15490 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:16.696442 15490 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:16.700862 15490 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7d97cdcb51184aebbb7288391f19ceaf: Generated new TSK 0
I20251023 20:12:16.701045 15490 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:12:16.727895 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.709807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.9.117.254:34609)
1023 20:12:16.709865 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.709867 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.710015 (+ 148us) server_negotiation.cc:408] Connection header received
1023 20:12:16.710122 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.710123 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.710147 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.710205 (+ 58us) server_negotiation.cc:227] Negotiated authn=JWT
1023 20:12:16.710534 (+ 329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:16.711089 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:16.711685 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:16.711865 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:16.711916 (+ 51us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1023 20:12:16.727581 (+ 15665us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1023 20:12:16.727660 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:16.727754 (+ 94us) server_negotiation.cc:300] Negotiation successful
1023 20:12:16.727789 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33,"thread_start_us":121,"threads_started":1}
I20251023 20:12:16.735843 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:16.732755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.9.117.254:34609)
1023 20:12:16.732807 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:16.732810 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:16.732903 (+ 93us) server_negotiation.cc:408] Connection header received
1023 20:12:16.732978 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:16.732980 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:16.732989 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:16.733044 (+ 55us) server_negotiation.cc:227] Negotiated authn=JWT
1023 20:12:16.733315 (+ 271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:16.734063 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:16.734865 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:16.735031 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:16.735053 (+ 22us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1023 20:12:16.735582 (+ 529us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1023 20:12:16.735738 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48448: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":26}
W20251023 20:12:16.735932 15482 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48448: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251023 20:12:16.736053 15514 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:16.732644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34609 (local address 127.0.0.1:48448)
1023 20:12:16.732827 (+ 183us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:16.732841 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:16.732912 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:16.733058 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:16.733058 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:16.733060 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1023 20:12:16.733244 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:16.733245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:16.734269 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:16.734271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:16.734750 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:16.734752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:16.734855 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:16.734863 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1023 20:12:16.735712 (+ 849us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1023 20:12:16.735818 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.117.254:34609: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":118,"threads_started":1}
I20251023 20:12:31.742467 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:31.739632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.9.117.254:34609)
1023 20:12:31.739834 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:31.739836 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:31.739878 (+ 42us) server_negotiation.cc:408] Connection header received
1023 20:12:31.739965 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:31.739966 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:31.739984 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:31.740052 (+ 68us) server_negotiation.cc:227] Negotiated authn=JWT
1023 20:12:31.740348 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.741022 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:31.741950 (+ 928us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.742098 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:31.742119 (+ 21us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1023 20:12:31.742298 (+ 179us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1023 20:12:31.742410 (+ 112us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34672: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":143,"threads_started":1}
W20251023 20:12:31.742532 15520 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34672: JWT verification failed: token expired
W20251023 20:12:31.742587 15519 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:31.739539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34609 (local address 127.0.0.1:34672)
1023 20:12:31.739780 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:31.739796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:31.739900 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:31.740067 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:31.740068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:31.740070 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1023 20:12:31.740270 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:31.740272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.741210 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:31.741212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:31.741875 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:31.741878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.741949 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:31.741955 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1023 20:12:31.742404 (+ 449us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1023 20:12:31.742520 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.117.254:34609: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
W20251023 20:12:31.744197 15520 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:31.743705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.9.117.254:34609)
1023 20:12:31.743800 (+ 95us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:31.743803 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:31.743833 (+ 30us) server_negotiation.cc:408] Connection header received
1023 20:12:31.743939 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:31.743940 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:31.743957 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:31.744014 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:31.744148 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34688: BlockingRecv error: recv got EOF from 127.0.0.1:34688 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20251023 20:12:31.747062 15530 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.9.117.254:34609
W20251023 20:12:31.747476 15520 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:31.746912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.9.117.254:34609)
1023 20:12:31.746956 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:31.746958 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:31.747043 (+ 85us) server_negotiation.cc:408] Connection header received
1023 20:12:31.747220 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:31.747221 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:31.747232 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:31.747290 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:31.747422 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34700: BlockingRecv error: recv got EOF from 127.0.0.1:34700 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20251023 20:12:31.747607 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 15411
2025-10-23T20:12:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15243 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251023 20:12:31.787573 9687 mini_oidc.cc:150] Starting JWKS server
I20251023 20:12:31.788360 9687 webserver.cc:492] Webserver started at https://127.0.0.1:36903/ using document root <none> and password file <none>
I20251023 20:12:31.788403 9687 mini_oidc.cc:186] Starting OIDC Discovery server
I20251023 20:12:31.788506 9687 webserver.cc:492] Webserver started at http://127.0.0.1:39043/ using document root <none> and password file <none>
2025-10-23T20:12:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:31Z Disabled control of system clock
I20251023 20:12:31.795840 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43653
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:43653
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36903/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/testchainca.cert with env {}
W20251023 20:12:31.881415 15536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:31.881646 15536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:31.881685 15536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:31.883286 15536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:31.883354 15536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:31.883376 15536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:31.883399 15536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:31.883419 15536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251023 20:12:31.883441 15536 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251023 20:12:31.883471 15536 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36903/jwks.json
I20251023 20:12:31.885332 15536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:43653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36903/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:31.885677 15536 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:31.885882 15536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:31.889470 15541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:31.889607 15536 server_base.cc:1047] running on GCE node
W20251023 20:12:31.889470 15544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:31.889519 15542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:31.890051 15536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:31.890316 15536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:31.891460 15536 hybrid_clock.cc:648] HybridClock initialized: now 1761250351891439 us; error 42 us; skew 500 ppm
I20251023 20:12:31.892688 15536 webserver.cc:492] Webserver started at http://127.9.117.254:46739/ using document root <none> and password file <none>
I20251023 20:12:31.892939 15536 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:31.892992 15536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:31.893115 15536 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:31.894109 15536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "5cb874e8925a44d9a8b095732c17b71f"
format_stamp: "Formatted at 2025-10-23 20:12:31 on dist-test-slave-x62p"
server_key: "a61b19ac8382ccaf105b33ef672f3c58"
server_key_iv: "ddf469f072d1f15c3375575bdf863376"
server_key_version: "encryptionkey@0"
I20251023 20:12:31.894429 15536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "5cb874e8925a44d9a8b095732c17b71f"
format_stamp: "Formatted at 2025-10-23 20:12:31 on dist-test-slave-x62p"
server_key: "a61b19ac8382ccaf105b33ef672f3c58"
server_key_iv: "ddf469f072d1f15c3375575bdf863376"
server_key_version: "encryptionkey@0"
I20251023 20:12:31.895707 15536 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:31.896394 15550 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:31.896595 15536 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:12:31.896674 15536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "5cb874e8925a44d9a8b095732c17b71f"
format_stamp: "Formatted at 2025-10-23 20:12:31 on dist-test-slave-x62p"
server_key: "a61b19ac8382ccaf105b33ef672f3c58"
server_key_iv: "ddf469f072d1f15c3375575bdf863376"
server_key_version: "encryptionkey@0"
I20251023 20:12:31.896775 15536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:31.929731 15536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:31.930100 15536 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:31.930234 15536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:31.934542 15536 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:43653
I20251023 20:12:31.934579 15602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:43653 every 8 connection(s)
I20251023 20:12:31.934926 15536 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:31.935638 15603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:31.938385 15603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f: Bootstrap starting.
I20251023 20:12:31.939003 15603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:31.939232 15603 log.cc:826] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:31.940044 15603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f: No bootstrap required, opened a new log
I20251023 20:12:31.940663 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 15536
I20251023 20:12:31.940783 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:31.940924 9687 external_mini_cluster.cc:1468] Setting key 8c313386a9a8e6853a7119c54d051672
I20251023 20:12:31.941699 15603 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb874e8925a44d9a8b095732c17b71f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43653 } }
I20251023 20:12:31.941881 15603 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:31.941918 15603 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb874e8925a44d9a8b095732c17b71f, State: Initialized, Role: FOLLOWER
I20251023 20:12:31.942040 15603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb874e8925a44d9a8b095732c17b71f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43653 } }
I20251023 20:12:31.942118 15603 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:31.942165 15603 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:31.942215 15603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:31.942951 15603 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb874e8925a44d9a8b095732c17b71f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43653 } }
I20251023 20:12:31.943055 15603 leader_election.cc:304] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb874e8925a44d9a8b095732c17b71f; no voters:
I20251023 20:12:31.943221 15603 leader_election.cc:290] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:31.943272 15608 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:31.943451 15608 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [term 1 LEADER]: Becoming Leader. State: Replica: 5cb874e8925a44d9a8b095732c17b71f, State: Running, Role: LEADER
I20251023 20:12:31.943473 15603 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:31.943647 15608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb874e8925a44d9a8b095732c17b71f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43653 } }
I20251023 20:12:31.945185 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb874e8925a44d9a8b095732c17b71f. Latest consensus state: current_term: 1 leader_uuid: "5cb874e8925a44d9a8b095732c17b71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb874e8925a44d9a8b095732c17b71f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43653 } } }
I20251023 20:12:31.945189 15610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb874e8925a44d9a8b095732c17b71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb874e8925a44d9a8b095732c17b71f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43653 } } }
I20251023 20:12:31.945513 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:31.945519 15610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f [sys.catalog]: This master's current role is: LEADER
W20251023 20:12:31.946298 15623 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:12:31.946357 15623 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:12:31.946442 15624 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:31.946645 15624 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:31.946686 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:31.941452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.9.117.254:43653)
1023 20:12:31.941707 (+ 255us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:31.941715 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:31.941742 (+ 27us) server_negotiation.cc:408] Connection header received
1023 20:12:31.941865 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:31.941871 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:31.941945 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:31.942093 (+ 148us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:31.942716 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.943709 (+ 993us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:31.944805 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.945063 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:31.945476 (+ 413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:31.945485 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:31.945486 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:31.946169 (+ 683us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:31.946199 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:31.946209 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:31.946292 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:31.946463 (+ 171us) server_negotiation.cc:300] Negotiation successful
1023 20:12:31.946548 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20251023 20:12:31.947561 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:12:31.947978 15624 catalog_manager.cc:1357] Generated new cluster ID: 28c3ca14387c4303822d1586b3b54b0b
I20251023 20:12:31.948036 15624 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:31.969928 15624 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:31.970489 15624 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:31.976533 15624 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5cb874e8925a44d9a8b095732c17b71f: Generated new TSK 0
I20251023 20:12:31.976765 15624 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251023 20:12:32.007364 15633 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251023 20:12:32.007470 15633 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251023 20:12:32.007823 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:31.992512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.9.117.254:43653)
1023 20:12:31.992574 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:31.992577 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:31.992658 (+ 81us) server_negotiation.cc:408] Connection header received
1023 20:12:31.992753 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:31.992755 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:31.992771 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:31.992848 (+ 77us) server_negotiation.cc:227] Negotiated authn=JWT
1023 20:12:31.993118 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.993956 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:31.994612 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.994803 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:31.994863 (+ 60us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1023 20:12:32.007525 (+ 12662us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1023 20:12:32.007733 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":29}
W20251023 20:12:32.007874 15632 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:31.992425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43653 (local address 127.0.0.1:39450)
1023 20:12:31.992577 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:31.992588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:31.992662 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:31.992860 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:31.992861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:31.992863 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1023 20:12:31.993043 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:31.993045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.994070 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:31.994071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:31.994526 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:31.994528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:31.994606 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:31.994613 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1023 20:12:32.007676 (+ 13063us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1023 20:12:32.007782 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.117.254:43653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20251023 20:12:32.007896 15607 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20251023 20:12:32.008283 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 15536
2025-10-23T20:12:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (258 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251023 20:12:32.046460 9687 mini_oidc.cc:150] Starting JWKS server
I20251023 20:12:32.047233 9687 webserver.cc:492] Webserver started at https://127.0.0.1:33295/ using document root <none> and password file <none>
I20251023 20:12:32.047276 9687 mini_oidc.cc:186] Starting OIDC Discovery server
I20251023 20:12:32.047382 9687 webserver.cc:492] Webserver started at http://127.0.0.1:38781/ using document root <none> and password file <none>
2025-10-23T20:12:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:32Z Disabled control of system clock
I20251023 20:12:32.055109 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:42461
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:42461
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33295/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251023 20:12:32.140203 15639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:32.140413 15639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:32.140440 15639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:32.142078 15639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:32.142132 15639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:32.142145 15639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:32.142158 15639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:32.142170 15639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251023 20:12:32.142182 15639 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251023 20:12:32.142194 15639 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33295/jwks.json
I20251023 20:12:32.143751 15639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:42461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33295/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:32.143996 15639 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:32.144261 15639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:32.147321 15647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:32.147436 15644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:32.147323 15645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:32.147702 15639 server_base.cc:1047] running on GCE node
I20251023 20:12:32.147980 15639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:32.148272 15639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:32.149436 15639 hybrid_clock.cc:648] HybridClock initialized: now 1761250352149402 us; error 42 us; skew 500 ppm
I20251023 20:12:32.151015 15639 webserver.cc:492] Webserver started at http://127.9.117.254:33937/ using document root <none> and password file <none>
I20251023 20:12:32.151274 15639 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:32.151324 15639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:32.151458 15639 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:32.152909 15639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "ef31a91ba6f44e1499fa25bff0e38a02"
format_stamp: "Formatted at 2025-10-23 20:12:32 on dist-test-slave-x62p"
server_key: "a577652eb6e795c5e0b93edcab7770c4"
server_key_iv: "9087239131c27ec42f7a093ab080615e"
server_key_version: "encryptionkey@0"
I20251023 20:12:32.153401 15639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "ef31a91ba6f44e1499fa25bff0e38a02"
format_stamp: "Formatted at 2025-10-23 20:12:32 on dist-test-slave-x62p"
server_key: "a577652eb6e795c5e0b93edcab7770c4"
server_key_iv: "9087239131c27ec42f7a093ab080615e"
server_key_version: "encryptionkey@0"
I20251023 20:12:32.155032 15639 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:12:32.155782 15653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:32.156005 15639 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:12:32.156083 15639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "ef31a91ba6f44e1499fa25bff0e38a02"
format_stamp: "Formatted at 2025-10-23 20:12:32 on dist-test-slave-x62p"
server_key: "a577652eb6e795c5e0b93edcab7770c4"
server_key_iv: "9087239131c27ec42f7a093ab080615e"
server_key_version: "encryptionkey@0"
I20251023 20:12:32.156150 15639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:32.185132 15639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:32.185482 15639 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:32.185621 15639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:32.191008 15639 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:42461
I20251023 20:12:32.191092 15705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:42461 every 8 connection(s)
I20251023 20:12:32.191615 15639 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:32.192612 15706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:32.196921 15706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02: Bootstrap starting.
I20251023 20:12:32.197593 15706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:32.197850 15706 log.cc:826] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:32.198740 15706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02: No bootstrap required, opened a new log
I20251023 20:12:32.200031 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 15639
I20251023 20:12:32.200135 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:32.200313 9687 external_mini_cluster.cc:1468] Setting key 8f5d4f049ccdbfefca9314f6815d5aee
I20251023 20:12:32.200716 15706 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef31a91ba6f44e1499fa25bff0e38a02" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42461 } }
I20251023 20:12:32.200925 15706 raft_consensus.cc:385] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:32.200963 15706 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef31a91ba6f44e1499fa25bff0e38a02, State: Initialized, Role: FOLLOWER
I20251023 20:12:32.201093 15706 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef31a91ba6f44e1499fa25bff0e38a02" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42461 } }
I20251023 20:12:32.201172 15706 raft_consensus.cc:399] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:32.201201 15706 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:32.201243 15706 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:32.201973 15706 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef31a91ba6f44e1499fa25bff0e38a02" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42461 } }
I20251023 20:12:32.202087 15706 leader_election.cc:304] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef31a91ba6f44e1499fa25bff0e38a02; no voters:
I20251023 20:12:32.202242 15706 leader_election.cc:290] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:32.202288 15711 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:32.202397 15711 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [term 1 LEADER]: Becoming Leader. State: Replica: ef31a91ba6f44e1499fa25bff0e38a02, State: Running, Role: LEADER
I20251023 20:12:32.202510 15706 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:32.202477 15711 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef31a91ba6f44e1499fa25bff0e38a02" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42461 } }
I20251023 20:12:32.202821 15711 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef31a91ba6f44e1499fa25bff0e38a02. Latest consensus state: current_term: 1 leader_uuid: "ef31a91ba6f44e1499fa25bff0e38a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef31a91ba6f44e1499fa25bff0e38a02" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42461 } } }
I20251023 20:12:32.202883 15711 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:32.203099 15715 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:32.203624 15715 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:32.204823 15712 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef31a91ba6f44e1499fa25bff0e38a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef31a91ba6f44e1499fa25bff0e38a02" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42461 } } }
I20251023 20:12:32.204900 15712 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:32.205627 15715 catalog_manager.cc:1357] Generated new cluster ID: 66ff3fc9977442439bdd1691f5e2ef65
I20251023 20:12:32.205687 15715 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:32.207562 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:32.201461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.9.117.254:42461)
1023 20:12:32.201709 (+ 248us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:32.201717 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:32.201743 (+ 26us) server_negotiation.cc:408] Connection header received
1023 20:12:32.201845 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:32.201849 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:32.201896 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:32.202040 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:32.202528 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.204934 (+ 2406us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:32.205765 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.206004 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:32.206870 (+ 866us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:32.206902 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:32.206903 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:32.207131 (+ 228us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:32.207167 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:32.207176 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:32.207260 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:32.207327 (+ 67us) server_negotiation.cc:300] Negotiation successful
1023 20:12:32.207390 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":117,"threads_started":1}
I20251023 20:12:32.208482 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:12:32.253086 15715 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:32.253935 15715 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:32.261613 15715 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ef31a91ba6f44e1499fa25bff0e38a02: Generated new TSK 0
I20251023 20:12:32.261829 15715 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251023 20:12:32.291762 15736 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251023 20:12:32.291852 15736 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251023 20:12:32.293059 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:32.274880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.9.117.254:42461)
1023 20:12:32.274931 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:32.274933 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:32.275020 (+ 87us) server_negotiation.cc:408] Connection header received
1023 20:12:32.275084 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:32.275085 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:32.275098 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:32.275162 (+ 64us) server_negotiation.cc:227] Negotiated authn=JWT
1023 20:12:32.275408 (+ 246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.276245 (+ 837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:32.276944 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.277132 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:32.277183 (+ 51us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1023 20:12:32.292775 (+ 15592us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1023 20:12:32.292990 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20251023 20:12:32.293133 15710 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251023 20:12:32.293445 15735 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:32.274773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:42461 (local address 127.0.0.1:57796)
1023 20:12:32.274939 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:32.274950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:32.275020 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:32.275177 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:32.275178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:32.275180 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1023 20:12:32.275349 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:32.275351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.276365 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:32.276367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:32.276866 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:32.276869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.276946 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:32.276952 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1023 20:12:32.292895 (+ 15943us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1023 20:12:32.293323 (+ 428us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.117.254:42461: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20251023 20:12:32.294129 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 15639
2025-10-23T20:12:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-10-23T20:12:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:32Z Disabled control of system clock
I20251023 20:12:32.341792 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:41147
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:41147
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/testchainca.cert with env {}
W20251023 20:12:32.423734 15740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:32.423928 15740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:32.423964 15740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:32.425431 15740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:32.425479 15740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:32.425503 15740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:32.425532 15740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:32.425551 15740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251023 20:12:32.425582 15740 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251023 20:12:32.425607 15740 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251023 20:12:32.427242 15740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:41147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:41147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:32.427508 15740 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:32.427686 15740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:32.430204 15745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:32.430210 15748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:32.430209 15746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:32.430411 15740 server_base.cc:1047] running on GCE node
I20251023 20:12:32.430629 15740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:32.430835 15740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:32.432034 15740 hybrid_clock.cc:648] HybridClock initialized: now 1761250352432019 us; error 33 us; skew 500 ppm
I20251023 20:12:32.433198 15740 webserver.cc:492] Webserver started at http://127.9.117.254:45899/ using document root <none> and password file <none>
I20251023 20:12:32.433411 15740 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:32.433454 15740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:32.433597 15740 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:32.434566 15740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "aae2e62b3133483790d5784a21747b53"
format_stamp: "Formatted at 2025-10-23 20:12:32 on dist-test-slave-x62p"
server_key: "4879f56c7d448f39014bb8f85b90a333"
server_key_iv: "966a0114644e7998b86d7596950c2296"
server_key_version: "encryptionkey@0"
I20251023 20:12:32.434891 15740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "aae2e62b3133483790d5784a21747b53"
format_stamp: "Formatted at 2025-10-23 20:12:32 on dist-test-slave-x62p"
server_key: "4879f56c7d448f39014bb8f85b90a333"
server_key_iv: "966a0114644e7998b86d7596950c2296"
server_key_version: "encryptionkey@0"
I20251023 20:12:32.436040 15740 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:12:32.436800 15754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:32.436985 15740 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:32.437065 15740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "aae2e62b3133483790d5784a21747b53"
format_stamp: "Formatted at 2025-10-23 20:12:32 on dist-test-slave-x62p"
server_key: "4879f56c7d448f39014bb8f85b90a333"
server_key_iv: "966a0114644e7998b86d7596950c2296"
server_key_version: "encryptionkey@0"
I20251023 20:12:32.437129 15740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:32.446784 15740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:32.447121 15740 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:32.447254 15740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:32.451530 15740 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:41147
I20251023 20:12:32.451555 15806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:41147 every 8 connection(s)
I20251023 20:12:32.451902 15740 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:32.452589 15807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:32.455148 15807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53: Bootstrap starting.
I20251023 20:12:32.455711 15807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:32.455920 15807 log.cc:826] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:32.455909 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 15740
I20251023 20:12:32.455988 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:32.456142 9687 external_mini_cluster.cc:1468] Setting key 6253df46576ea5132b6192d271ba8919
I20251023 20:12:32.456808 15807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53: No bootstrap required, opened a new log
I20251023 20:12:32.458284 15807 raft_consensus.cc:359] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae2e62b3133483790d5784a21747b53" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41147 } }
I20251023 20:12:32.458438 15807 raft_consensus.cc:385] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:32.458472 15807 raft_consensus.cc:740] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae2e62b3133483790d5784a21747b53, State: Initialized, Role: FOLLOWER
I20251023 20:12:32.458586 15807 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae2e62b3133483790d5784a21747b53" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41147 } }
I20251023 20:12:32.458668 15807 raft_consensus.cc:399] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:32.458691 15807 raft_consensus.cc:493] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:32.458725 15807 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:32.459446 15807 raft_consensus.cc:515] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae2e62b3133483790d5784a21747b53" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41147 } }
I20251023 20:12:32.459563 15807 leader_election.cc:304] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae2e62b3133483790d5784a21747b53; no voters:
I20251023 20:12:32.459719 15807 leader_election.cc:290] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:32.459822 15812 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:32.459939 15807 sys_catalog.cc:565] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:32.460002 15812 raft_consensus.cc:697] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [term 1 LEADER]: Becoming Leader. State: Replica: aae2e62b3133483790d5784a21747b53, State: Running, Role: LEADER
I20251023 20:12:32.460091 15812 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae2e62b3133483790d5784a21747b53" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41147 } }
I20251023 20:12:32.460389 15813 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aae2e62b3133483790d5784a21747b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae2e62b3133483790d5784a21747b53" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41147 } } }
I20251023 20:12:32.460455 15813 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:32.461043 15814 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae2e62b3133483790d5784a21747b53. Latest consensus state: current_term: 1 leader_uuid: "aae2e62b3133483790d5784a21747b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae2e62b3133483790d5784a21747b53" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 41147 } } }
I20251023 20:12:32.461194 15814 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53 [sys.catalog]: This master's current role is: LEADER
W20251023 20:12:32.462716 15828 catalog_manager.cc:1568] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:12:32.462785 15828 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:12:32.462852 15818 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:32.462918 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:32.456694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.9.117.254:41147)
1023 20:12:32.456973 (+ 279us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:32.456982 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:32.457006 (+ 24us) server_negotiation.cc:408] Connection header received
1023 20:12:32.458248 (+ 1242us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:32.458259 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:32.458334 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:32.458483 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:32.458952 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.460136 (+ 1184us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:32.461538 (+ 1402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.461807 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:32.461964 (+ 157us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:32.461973 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:32.461974 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:12:32.462216 (+ 242us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:32.462254 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:12:32.462264 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:32.462613 (+ 349us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:32.462654 (+ 41us) server_negotiation.cc:300] Negotiation successful
1023 20:12:32.462712 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":129,"threads_started":1}
I20251023 20:12:32.463027 15818 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:32.463898 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:12:32.464457 15818 catalog_manager.cc:1357] Generated new cluster ID: 466842382d0744128ee80db2046d2fcd
I20251023 20:12:32.464516 15818 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:32.478055 15818 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:32.478694 15818 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:32.488497 15818 catalog_manager.cc:6022] T 00000000000000000000000000000000 P aae2e62b3133483790d5784a21747b53: Generated new TSK 0
I20251023 20:12:32.488775 15818 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:12:32.529839 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:32.496543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.9.117.254:41147)
1023 20:12:32.496602 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:32.496604 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:32.496761 (+ 157us) server_negotiation.cc:408] Connection header received
1023 20:12:32.496874 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:32.496875 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:32.496887 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:32.496939 (+ 52us) server_negotiation.cc:227] Negotiated authn=JWT
1023 20:12:32.497205 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.497742 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:32.498334 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.498500 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:32.498591 (+ 91us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1023 20:12:32.529491 (+ 30900us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1023 20:12:32.529747 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48456: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":31}
W20251023 20:12:32.529932 15811 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48456: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20251023 20:12:32.529942 15836 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:32.496470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:41147 (local address 127.0.0.1:48456)
1023 20:12:32.496663 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:32.496673 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:32.496775 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:32.496952 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:32.496953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:32.496955 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1023 20:12:32.497138 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:32.497140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.497830 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:32.497832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:32.498247 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:32.498249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:32.498332 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:32.498338 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1023 20:12:32.529667 (+ 31329us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1023 20:12:32.529822 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.117.254:41147: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
I20251023 20:12:32.530226 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 15740
2025-10-23T20:12:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (213 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (92 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:32 dist-test-slave-x62p krb5kdc[15848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:32 dist-test-slave-x62p krb5kdc[15848](info): set up 2 sockets
Oct 23 20:12:32 dist-test-slave-x62p krb5kdc[15848](info): commencing operation
krb5kdc: starting...
W20251023 20:12:34.709460 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:34 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:34.852905 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43761
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:43761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:34.935209 15864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:34.935405 15864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:34.935444 15864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:34.936976 15864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:34.937029 15864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:34.937053 15864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:34.937072 15864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:34.937094 15864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:34.938797 15864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:43761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:34.939103 15864 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:34.939304 15864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:34.942076 15869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:34.942121 15870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:34.942175 15864 server_base.cc:1047] running on GCE node
W20251023 20:12:34.942121 15872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:34.942514 15864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:34.942775 15864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:34.943912 15864 hybrid_clock.cc:648] HybridClock initialized: now 1761250354943896 us; error 34 us; skew 500 ppm
Oct 23 20:12:34 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:34.946466 15864 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:34.947052 15864 webserver.cc:492] Webserver started at http://127.9.117.254:32871/ using document root <none> and password file <none>
I20251023 20:12:34.947256 15864 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:34.947297 15864 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:34.947410 15864 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:34.948411 15864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "08fc0b5b0e2c4c6282f94084b9527a24"
format_stamp: "Formatted at 2025-10-23 20:12:34 on dist-test-slave-x62p"
server_key: "852f548a9ebac4c06fd371a2839acbfd"
server_key_iv: "9320f7cd5e871e0c3dd808cf704566ff"
server_key_version: "encryptionkey@0"
I20251023 20:12:34.948760 15864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "08fc0b5b0e2c4c6282f94084b9527a24"
format_stamp: "Formatted at 2025-10-23 20:12:34 on dist-test-slave-x62p"
server_key: "852f548a9ebac4c06fd371a2839acbfd"
server_key_iv: "9320f7cd5e871e0c3dd808cf704566ff"
server_key_version: "encryptionkey@0"
I20251023 20:12:34.950013 15864 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:34.950766 15879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:34.950973 15864 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:34.951048 15864 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "08fc0b5b0e2c4c6282f94084b9527a24"
format_stamp: "Formatted at 2025-10-23 20:12:34 on dist-test-slave-x62p"
server_key: "852f548a9ebac4c06fd371a2839acbfd"
server_key_iv: "9320f7cd5e871e0c3dd808cf704566ff"
server_key_version: "encryptionkey@0"
I20251023 20:12:34.951113 15864 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:34.960788 15864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:34.962713 15864 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:34.962869 15864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:34.967199 15864 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:43761
I20251023 20:12:34.967243 15931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:43761 every 8 connection(s)
I20251023 20:12:34.967593 15864 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:34.968292 15932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:34.971029 15932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24: Bootstrap starting.
I20251023 20:12:34.971655 15932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:34.971915 15932 log.cc:826] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:34.972667 15932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24: No bootstrap required, opened a new log
I20251023 20:12:34.973948 15932 raft_consensus.cc:359] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc0b5b0e2c4c6282f94084b9527a24" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43761 } }
I20251023 20:12:34.974084 15932 raft_consensus.cc:385] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:34.974105 15932 raft_consensus.cc:740] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fc0b5b0e2c4c6282f94084b9527a24, State: Initialized, Role: FOLLOWER
I20251023 20:12:34.974205 15932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc0b5b0e2c4c6282f94084b9527a24" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43761 } }
I20251023 20:12:34.974256 15932 raft_consensus.cc:399] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:34.974287 15932 raft_consensus.cc:493] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:34.974331 15932 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:34.974905 15932 raft_consensus.cc:515] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc0b5b0e2c4c6282f94084b9527a24" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43761 } }
I20251023 20:12:34.975018 15932 leader_election.cc:304] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08fc0b5b0e2c4c6282f94084b9527a24; no voters:
I20251023 20:12:34.975175 15932 leader_election.cc:290] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:34.975272 15935 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:34.975414 15932 sys_catalog.cc:565] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:34.975425 15935 raft_consensus.cc:697] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [term 1 LEADER]: Becoming Leader. State: Replica: 08fc0b5b0e2c4c6282f94084b9527a24, State: Running, Role: LEADER
I20251023 20:12:34.975574 15935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc0b5b0e2c4c6282f94084b9527a24" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43761 } }
I20251023 20:12:34.975906 15936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08fc0b5b0e2c4c6282f94084b9527a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc0b5b0e2c4c6282f94084b9527a24" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43761 } } }
I20251023 20:12:34.976002 15936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:34.975919 15937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08fc0b5b0e2c4c6282f94084b9527a24. Latest consensus state: current_term: 1 leader_uuid: "08fc0b5b0e2c4c6282f94084b9527a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc0b5b0e2c4c6282f94084b9527a24" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43761 } } }
I20251023 20:12:34.976135 15937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:34.976301 15943 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:34.977106 15943 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:34.977618 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 15864
I20251023 20:12:34.977732 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:34.977928 9687 external_mini_cluster.cc:1468] Setting key af057ea0b490eeea45f95b88a9b0e1d7
I20251023 20:12:34.978811 15943 catalog_manager.cc:1357] Generated new cluster ID: ec15de6f11ab4a729e3ee6fc1b7949b6
I20251023 20:12:34.978864 15943 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:12:34 dist-test-slave-x62p krb5kdc[15848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:34.986743 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:34.978405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.9.117.254:43761)
1023 20:12:34.978564 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:34.978576 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:34.978614 (+ 38us) server_negotiation.cc:408] Connection header received
1023 20:12:34.978743 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:34.978749 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:34.978804 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:34.978940 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:34.979828 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:34.980609 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:34.981224 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:34.981866 (+ 642us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:34.983557 (+ 1691us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:34.983564 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:34.983564 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:34.983585 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:34.985385 (+ 1800us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:34.985727 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:34.985729 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:34.985730 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:34.985785 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:34.985936 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:34.985937 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:34.985938 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:34.986152 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:34.986303 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:34.986427 (+ 124us) server_negotiation.cc:300] Negotiation successful
1023 20:12:34.986569 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20251023 20:12:34.997059 15943 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:34.997905 15943 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:35.005167 15943 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 08fc0b5b0e2c4c6282f94084b9527a24: Generated new TSK 0
I20251023 20:12:35.005393 15943 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:35.059216 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43761
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:35.142525 15960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:35.142706 15960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:35.142731 15960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:35.144233 15960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:35.144290 15960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:35.144320 15960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:35.145953 15960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:35.146209 15960 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:35.146402 15960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:35.149173 15966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:35.149189 15968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:35.149240 15965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:35.149374 15960 server_base.cc:1047] running on GCE node
I20251023 20:12:35.149550 15960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:35.149783 15960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:35.150921 15960 hybrid_clock.cc:648] HybridClock initialized: now 1761250355150906 us; error 37 us; skew 500 ppm
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.153244 15960 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:12:35.153822 15960 webserver.cc:492] Webserver started at http://127.9.117.193:40053/ using document root <none> and password file <none>
I20251023 20:12:35.154012 15960 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:35.154049 15960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:35.154160 15960 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:35.155195 15960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "7d0c8abff02a4e1ca1e6c9c9a667d1f2"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "d54086adcf3cf1961a73ae7db3106866"
server_key_iv: "48e5dc41e971a03601e97ed1df07ccf9"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.155498 15960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "7d0c8abff02a4e1ca1e6c9c9a667d1f2"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "d54086adcf3cf1961a73ae7db3106866"
server_key_iv: "48e5dc41e971a03601e97ed1df07ccf9"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.156715 15960 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:35.157562 15975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.157784 15960 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:35.157860 15960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "7d0c8abff02a4e1ca1e6c9c9a667d1f2"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "d54086adcf3cf1961a73ae7db3106866"
server_key_iv: "48e5dc41e971a03601e97ed1df07ccf9"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.157927 15960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:35.184846 15960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:35.186995 15960 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:35.187147 15960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:35.187374 15960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:35.187721 15960 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:35.187754 15960 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.187793 15960 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:35.187819 15960 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.194082 15960 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:36985
I20251023 20:12:35.194169 16088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:36985 every 8 connection(s)
I20251023 20:12:35.194456 15960 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:35.203683 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.195152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:50503 (local address 127.9.117.254:43761)
1023 20:12:35.195196 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.195199 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.195356 (+ 157us) server_negotiation.cc:408] Connection header received
1023 20:12:35.195860 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.195861 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.195877 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.195942 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.196694 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.197213 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.198231 (+ 1018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.198378 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.200661 (+ 2283us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:35.200668 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:35.200669 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:35.200677 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:35.202273 (+ 1596us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.202694 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.202696 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.202696 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.202727 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.202994 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.202995 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.202996 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.203109 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:35.203177 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:35.203468 (+ 291us) server_negotiation.cc:300] Negotiation successful
1023 20:12:35.203595 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251023 20:12:35.203797 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.195101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43761 (local address 127.9.117.193:50503)
1023 20:12:35.195256 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:35.195287 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:35.195773 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:35.196000 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:35.196004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:35.196316 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:35.196620 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:35.196624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.197297 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.197298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:35.198106 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:35.198115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.198243 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.199022 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:35.199051 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:35.200537 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:35.202371 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:35.202379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:35.202382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:35.202597 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:35.202821 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:35.202822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:35.202822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:35.202903 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:35.203177 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:35.203181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:35.203331 (+ 150us) client_negotiation.cc:769] Sending connection context
1023 20:12:35.203435 (+ 104us) client_negotiation.cc:240] Negotiation successful
1023 20:12:35.203599 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20251023 20:12:35.204376 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 15960
I20251023 20:12:35.204396 16089 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43761
I20251023 20:12:35.204491 16089 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:35.204495 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:35.204668 9687 external_mini_cluster.cc:1468] Setting key ff6aac87e516dbbc30598457993a424c
I20251023 20:12:35.204783 16089 heartbeater.cc:507] Master 127.9.117.254:43761 requested a full tablet report, sending...
I20251023 20:12:35.205341 15896 ts_manager.cc:194] Registered new tserver with Master: 7d0c8abff02a4e1ca1e6c9c9a667d1f2 (127.9.117.193:36985)
I20251023 20:12:35.206312 15896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:50503
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:35.256120 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43761
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:35.341166 16096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:35.341372 16096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:35.341414 16096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:35.342927 16096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:35.342989 16096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:35.343039 16096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:35.344678 16096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:35.345000 16096 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:35.345204 16096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:35.348055 16101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:35.348066 16102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:35.348090 16104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:35.348830 16096 server_base.cc:1047] running on GCE node
I20251023 20:12:35.348996 16096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:35.349201 16096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:35.350389 16096 hybrid_clock.cc:648] HybridClock initialized: now 1761250355350376 us; error 35 us; skew 500 ppm
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.353035 16096 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:12:35.353641 16096 webserver.cc:492] Webserver started at http://127.9.117.194:35181/ using document root <none> and password file <none>
I20251023 20:12:35.353852 16096 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:35.353888 16096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:35.353981 16096 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:35.355134 16096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "74a10a42dec741399196024e4b44d339"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "376d61c1b301583e10c049c6a636279d"
server_key_iv: "f71135ecb74c68c3cb1a90656fa8792e"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.355439 16096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "74a10a42dec741399196024e4b44d339"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "376d61c1b301583e10c049c6a636279d"
server_key_iv: "f71135ecb74c68c3cb1a90656fa8792e"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.357000 16096 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:12:35.357867 16111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.358093 16096 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:35.358171 16096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "74a10a42dec741399196024e4b44d339"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "376d61c1b301583e10c049c6a636279d"
server_key_iv: "f71135ecb74c68c3cb1a90656fa8792e"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.358248 16096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:35.384496 16096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:35.386866 16096 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:35.386996 16096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:35.387276 16096 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:35.387658 16096 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:35.387698 16096 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.387738 16096 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:35.387760 16096 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.394165 16096 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:43943
I20251023 20:12:35.394218 16224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:43943 every 8 connection(s)
I20251023 20:12:35.394551 16096 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:35.401422 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 16096
I20251023 20:12:35.401530 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:35.401764 9687 external_mini_cluster.cc:1468] Setting key 1d474beb992b72143aea63ec8c1c0db7
I20251023 20:12:35.403816 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.395336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:33785 (local address 127.9.117.254:43761)
1023 20:12:35.395422 (+ 86us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.395429 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.395584 (+ 155us) server_negotiation.cc:408] Connection header received
1023 20:12:35.396138 (+ 554us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.396140 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.396150 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.396222 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.397153 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.397691 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.398561 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.398697 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.400856 (+ 2159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:35.400861 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:35.400861 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:35.400869 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:35.402522 (+ 1653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.402930 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.402931 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.402932 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.402960 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.403141 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.403142 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.403142 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.403272 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:35.403334 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:35.403578 (+ 244us) server_negotiation.cc:300] Negotiation successful
1023 20:12:35.403685 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251023 20:12:35.403863 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.395288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43761 (local address 127.9.117.194:33785)
1023 20:12:35.395477 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:35.395510 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:35.396030 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:35.396338 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:35.396343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:35.396712 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:35.397041 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:35.397044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.397782 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.397783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:35.398485 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:35.398487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.398628 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.399190 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:35.399211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:35.400700 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:35.402622 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:35.402628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:35.402632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:35.402853 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:35.403022 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:35.403023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:35.403023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:35.403084 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:35.403330 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:35.403333 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:35.403456 (+ 123us) client_negotiation.cc:769] Sending connection context
1023 20:12:35.403545 (+ 89us) client_negotiation.cc:240] Negotiation successful
1023 20:12:35.403680 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20251023 20:12:35.404552 16225 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43761
I20251023 20:12:35.404647 16225 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:35.404894 16225 heartbeater.cc:507] Master 127.9.117.254:43761 requested a full tablet report, sending...
I20251023 20:12:35.405349 15896 ts_manager.cc:194] Registered new tserver with Master: 74a10a42dec741399196024e4b44d339 (127.9.117.194:43943)
I20251023 20:12:35.405894 15896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:33785
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:35.448033 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43761
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:35.530838 16232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:35.531021 16232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:35.531059 16232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:35.532459 16232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:35.532509 16232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:35.532552 16232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:35.534149 16232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:35.534422 16232 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:35.534626 16232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:35.537240 16240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:35.537389 16232 server_base.cc:1047] running on GCE node
W20251023 20:12:35.537249 16237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:35.537246 16238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:35.537737 16232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:35.537956 16232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:35.539088 16232 hybrid_clock.cc:648] HybridClock initialized: now 1761250355539074 us; error 32 us; skew 500 ppm
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.541451 16232 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:12:35.541975 16232 webserver.cc:492] Webserver started at http://127.9.117.195:39519/ using document root <none> and password file <none>
I20251023 20:12:35.542163 16232 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:35.542203 16232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:35.542317 16232 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:35.543315 16232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "81e3f313c5cc485899da69e7a3c29a92"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "7f1115666b2ec1c302efe46014085270"
server_key_iv: "8498606bd0a9a4a64859896da7485b7e"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.543623 16232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "81e3f313c5cc485899da69e7a3c29a92"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "7f1115666b2ec1c302efe46014085270"
server_key_iv: "8498606bd0a9a4a64859896da7485b7e"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.544930 16232 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:35.545665 16247 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.545866 16232 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:35.545948 16232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "81e3f313c5cc485899da69e7a3c29a92"
format_stamp: "Formatted at 2025-10-23 20:12:35 on dist-test-slave-x62p"
server_key: "7f1115666b2ec1c302efe46014085270"
server_key_iv: "8498606bd0a9a4a64859896da7485b7e"
server_key_version: "encryptionkey@0"
I20251023 20:12:35.546013 16232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:35.568491 16232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:35.570533 16232 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:35.570680 16232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:35.570932 16232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:35.571264 16232 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:35.571297 16232 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.571334 16232 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:35.571362 16232 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:35.577699 16232 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:46477
I20251023 20:12:35.577742 16360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:46477 every 8 connection(s)
I20251023 20:12:35.578070 16232 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:12:35.582968 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 16232
I20251023 20:12:35.583069 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:35.583253 9687 external_mini_cluster.cc:1468] Setting key 553b3f4c4104ebe928c5ce4a3e22785a
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:35.587013 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.578831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:49983 (local address 127.9.117.254:43761)
1023 20:12:35.578881 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.578883 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.579032 (+ 149us) server_negotiation.cc:408] Connection header received
1023 20:12:35.579559 (+ 527us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.579561 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.579571 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.579616 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.580460 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.581029 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.581918 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.582059 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.584354 (+ 2295us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:35.584358 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:35.584358 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:35.584367 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:35.585777 (+ 1410us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.586238 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.586239 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.586240 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.586268 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.586442 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.586442 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.586443 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.586537 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:35.586595 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:35.586849 (+ 254us) server_negotiation.cc:300] Negotiation successful
1023 20:12:35.586942 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251023 20:12:35.587121 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.578785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43761 (local address 127.9.117.195:49983)
1023 20:12:35.578921 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:35.578950 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:35.579465 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:35.579693 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:35.579697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:35.580036 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:35.580336 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:35.580340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.581115 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.581116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:35.581822 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:35.581824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.581956 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.582509 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:35.582531 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:35.584261 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:35.585868 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:35.585874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:35.585878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:35.586089 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:35.586332 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:35.586333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:35.586333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:35.586395 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:35.586584 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:35.586586 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:35.586715 (+ 129us) client_negotiation.cc:769] Sending connection context
1023 20:12:35.586806 (+ 91us) client_negotiation.cc:240] Negotiation successful
1023 20:12:35.586940 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20251023 20:12:35.587687 16361 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43761
I20251023 20:12:35.587798 16361 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:35.588034 16361 heartbeater.cc:507] Master 127.9.117.254:43761 requested a full tablet report, sending...
I20251023 20:12:35.588411 15896 ts_manager.cc:194] Registered new tserver with Master: 81e3f313c5cc485899da69e7a3c29a92 (127.9.117.195:46477)
I20251023 20:12:35.588855 15896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:49983
I20251023 20:12:35.595369 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/test-admin.keytab.
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.612039 9687 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251023 20:12:35.613796 16368 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:35.612992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.9.117.193:36985)
1023 20:12:35.613165 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.613174 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.613187 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:12:35.613218 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.613220 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.613234 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.613300 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.613728 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59106: BlockingRecv error: recv got EOF from 127.0.0.1:59106 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
W20251023 20:12:35.613989 9687 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5cc' of type FILE
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.615170 9687 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.616300 9687 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:12:35.623703 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.616929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.9.117.193:36985)
1023 20:12:35.616988 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.616990 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.617045 (+ 55us) server_negotiation.cc:408] Connection header received
1023 20:12:35.617095 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.617096 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.617107 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.617159 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.617696 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.618403 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.618949 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.619149 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.621129 (+ 1980us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:35.621141 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:35.621141 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:35.621149 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:35.622870 (+ 1721us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.623131 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.623134 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.623134 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.623166 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.623266 (+ 100us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.623267 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.623267 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.623361 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:35.623476 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:35.623510 (+ 34us) server_negotiation.cc:300] Negotiation successful
1023 20:12:35.623583 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20251023 20:12:35.625735 16368 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:35.625096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.9.117.193:36985)
1023 20:12:35.625137 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.625139 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.625188 (+ 49us) server_negotiation.cc:408] Connection header received
1023 20:12:35.625231 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.625232 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.625241 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.625282 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.625664 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59130: BlockingRecv error: recv got EOF from 127.0.0.1:59130 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20251023 20:12:35.625777 9687 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5cc)
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.626842 9687 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.627751 9687 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:12:35.634830 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.628355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.9.117.193:36985)
1023 20:12:35.628430 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.628432 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.628442 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:35.628475 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.628476 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.628484 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.628555 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.629121 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.629654 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.630192 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.630384 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.632364 (+ 1980us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:35.632370 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:35.632370 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:35.632378 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:35.633944 (+ 1566us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.634244 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.634245 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.634245 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.634261 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.634364 (+ 103us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.634364 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.634364 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.634467 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:35.634546 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:35.634630 (+ 84us) server_negotiation.cc:300] Negotiation successful
1023 20:12:35.634698 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20251023 20:12:35.636464 16368 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:35.635970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.9.117.193:36985)
1023 20:12:35.636010 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.636011 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.636108 (+ 97us) server_negotiation.cc:408] Connection header received
1023 20:12:35.636145 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.636146 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.636154 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.636199 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.636419 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59140: BlockingRecv error: recv got EOF from 127.0.0.1:59140 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20251023 20:12:35.636554 9687 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761250300132822-9687-0/krb5kdc/krb5cc' of type FILE
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.637585 9687 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:35.638427 9687 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[15848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
I20251023 20:12:35.645238 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:35.638991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.9.117.193:36985)
1023 20:12:35.639046 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:35.639048 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:35.639057 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:35.639112 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:35.639113 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:35.639120 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:35.639165 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:35.639629 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.640092 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:35.640591 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:35.640764 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:35.642807 (+ 2043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:35.642811 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:35.642811 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:35.642818 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:35.644285 (+ 1467us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.644591 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.644591 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.644592 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.644609 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:35.644775 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:35.644776 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:35.644776 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:35.644868 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:35.644947 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:35.645078 (+ 131us) server_negotiation.cc:300] Negotiation successful
1023 20:12:35.645147 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251023 20:12:35.645928 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 15960
I20251023 20:12:35.650799 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 16096
I20251023 20:12:35.656641 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 16232
I20251023 20:12:35.661867 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 15864
2025-10-23T20:12:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2948 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[16381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[16381](info): set up 2 sockets
Oct 23 20:12:35 dist-test-slave-x62p krb5kdc[16381](info): commencing operation
krb5kdc: starting...
W20251023 20:12:37.665434 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.983s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:37 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:37.812258 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33847
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:37.895201 16397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:37.895416 16397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:37.895457 16397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:37.897028 16397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:37.897080 16397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:37.897104 16397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:37.897126 16397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:37.897146 16397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:37.898880 16397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:37.899154 16397 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:37.899348 16397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:37.902100 16405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:37.902148 16403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:37.902139 16402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:37.902313 16397 server_base.cc:1047] running on GCE node
I20251023 20:12:37.902509 16397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:37.902768 16397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:37.903908 16397 hybrid_clock.cc:648] HybridClock initialized: now 1761250357903904 us; error 42 us; skew 500 ppm
Oct 23 20:12:37 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250357, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:37.906383 16397 init.cc:377] Logged in from keytab as oryx/127.9.117.254@KRBTEST.COM (short username oryx)
I20251023 20:12:37.906941 16397 webserver.cc:492] Webserver started at http://127.9.117.254:37431/ using document root <none> and password file <none>
I20251023 20:12:37.907114 16397 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:37.907146 16397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:37.907238 16397 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:37.908178 16397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "f53480ec5ac94aed91148cbc54a3e99f"
format_stamp: "Formatted at 2025-10-23 20:12:37 on dist-test-slave-x62p"
server_key: "491bdbdc407ecbf9c36794394e54fbe4"
server_key_iv: "2485f7f7a49c18b6066dc10ac51ef26a"
server_key_version: "encryptionkey@0"
I20251023 20:12:37.908452 16397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "f53480ec5ac94aed91148cbc54a3e99f"
format_stamp: "Formatted at 2025-10-23 20:12:37 on dist-test-slave-x62p"
server_key: "491bdbdc407ecbf9c36794394e54fbe4"
server_key_iv: "2485f7f7a49c18b6066dc10ac51ef26a"
server_key_version: "encryptionkey@0"
I20251023 20:12:37.909688 16397 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:37.910348 16412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:37.910531 16397 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:12:37.910605 16397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "f53480ec5ac94aed91148cbc54a3e99f"
format_stamp: "Formatted at 2025-10-23 20:12:37 on dist-test-slave-x62p"
server_key: "491bdbdc407ecbf9c36794394e54fbe4"
server_key_iv: "2485f7f7a49c18b6066dc10ac51ef26a"
server_key_version: "encryptionkey@0"
I20251023 20:12:37.910662 16397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:37.930258 16397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:37.937664 16397 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:37.937788 16397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:37.942006 16397 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:33847
I20251023 20:12:37.942061 16464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:33847 every 8 connection(s)
I20251023 20:12:37.942377 16397 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:37.943078 16465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:37.945816 16465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: Bootstrap starting.
I20251023 20:12:37.946440 16465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:37.946697 16465 log.cc:826] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:37.947031 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 16397
I20251023 20:12:37.947149 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:37.947311 9687 external_mini_cluster.cc:1468] Setting key 6331f1f66a54e1d3e94dbe13647ed1ce
I20251023 20:12:37.947456 16465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: No bootstrap required, opened a new log
I20251023 20:12:37.949308 16465 raft_consensus.cc:359] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } }
I20251023 20:12:37.949440 16465 raft_consensus.cc:385] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:37.949460 16465 raft_consensus.cc:740] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53480ec5ac94aed91148cbc54a3e99f, State: Initialized, Role: FOLLOWER
I20251023 20:12:37.949551 16465 consensus_queue.cc:260] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } }
I20251023 20:12:37.949640 16465 raft_consensus.cc:399] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:37.949679 16465 raft_consensus.cc:493] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:37.949726 16465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:37.950364 16465 raft_consensus.cc:515] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } }
I20251023 20:12:37.950467 16465 leader_election.cc:304] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f53480ec5ac94aed91148cbc54a3e99f; no voters:
I20251023 20:12:37.950637 16465 leader_election.cc:290] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:37.950686 16470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:37.950850 16470 raft_consensus.cc:697] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 1 LEADER]: Becoming Leader. State: Replica: f53480ec5ac94aed91148cbc54a3e99f, State: Running, Role: LEADER
I20251023 20:12:37.950945 16465 sys_catalog.cc:565] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:37.951045 16470 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } }
I20251023 20:12:37.951406 16470 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f53480ec5ac94aed91148cbc54a3e99f. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } }
I20251023 20:12:37.951449 16472 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } }
I20251023 20:12:37.951606 16470 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:37.951618 16472 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:37.951901 16479 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:37.952894 16479 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 23 20:12:37 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:37.954907 16479 catalog_manager.cc:1357] Generated new cluster ID: b9ef92cf951f4d268d65e7afaf5f4080
I20251023 20:12:37.954972 16479 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:37.958981 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:37.947852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.9.117.254:33847)
1023 20:12:37.948040 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:37.948052 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:37.948080 (+ 28us) server_negotiation.cc:408] Connection header received
1023 20:12:37.948622 (+ 542us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:37.948633 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:37.948700 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:37.948881 (+ 181us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:37.949652 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:37.950456 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:37.951213 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:37.951613 (+ 400us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:37.955082 (+ 3469us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:37.955109 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:37.955110 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:37.955137 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:37.957434 (+ 2297us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:37.957875 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:37.957879 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:37.957879 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:37.957949 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:37.958106 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:37.958107 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:37.958107 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:37.958384 (+ 277us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:37.958532 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:37.958644 (+ 112us) server_negotiation.cc:300] Negotiation successful
1023 20:12:37.958783 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20251023 20:12:37.971038 16479 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:37.971670 16479 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:37.980006 16479 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: Generated new TSK 0
I20251023 20:12:37.980261 16479 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:38.033808 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:33847
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:38.121136 16493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:38.121331 16493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:38.121359 16493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:38.122872 16493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:38.122924 16493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:38.122952 16493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:38.124578 16493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:38.124858 16493 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:38.125061 16493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:38.127754 16501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.127885 16499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.127907 16498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:38.128007 16493 server_base.cc:1047] running on GCE node
I20251023 20:12:38.128228 16493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:38.128433 16493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:38.129575 16493 hybrid_clock.cc:648] HybridClock initialized: now 1761250358129556 us; error 32 us; skew 500 ppm
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:38.132026 16493 init.cc:377] Logged in from keytab as oryx/127.9.117.193@KRBTEST.COM (short username oryx)
I20251023 20:12:38.132575 16493 webserver.cc:492] Webserver started at http://127.9.117.193:40521/ using document root <none> and password file <none>
I20251023 20:12:38.132817 16493 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:38.132859 16493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:38.132973 16493 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:38.133936 16493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "7b5a4db60789429ba2b82ab3f1bff442"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "2f9a3941f72dc662c782cfdf87f8b4ec"
server_key_iv: "ed62ed383f2bb0706eac094c63cd9da1"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.134243 16493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "7b5a4db60789429ba2b82ab3f1bff442"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "2f9a3941f72dc662c782cfdf87f8b4ec"
server_key_iv: "ed62ed383f2bb0706eac094c63cd9da1"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.135426 16493 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:12:38.136101 16508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.136274 16493 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:12:38.136346 16493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "7b5a4db60789429ba2b82ab3f1bff442"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "2f9a3941f72dc662c782cfdf87f8b4ec"
server_key_iv: "ed62ed383f2bb0706eac094c63cd9da1"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.136406 16493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:38.154747 16493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:38.156847 16493 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:38.157007 16493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:38.157241 16493 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:38.157581 16493 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:38.157634 16493 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.157671 16493 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:38.157703 16493 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.163786 16493 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:39439
I20251023 20:12:38.163882 16621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:39439 every 8 connection(s)
I20251023 20:12:38.164177 16493 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:38.168517 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 16493
I20251023 20:12:38.168618 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:38.168810 9687 external_mini_cluster.cc:1468] Setting key 05b0136bdd07ec48eda8e5f5add29ec6
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:38.173821 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.165037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:55599 (local address 127.9.117.254:33847)
1023 20:12:38.165120 (+ 83us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.165122 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.165244 (+ 122us) server_negotiation.cc:408] Connection header received
1023 20:12:38.165786 (+ 542us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.165788 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.165799 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.165874 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.166767 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.167274 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.167839 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.167944 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.170808 (+ 2864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.170818 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.170819 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.170826 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.172392 (+ 1566us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.172921 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.172924 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.172924 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.172951 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.173148 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.173149 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.173149 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.173256 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.173319 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.173595 (+ 276us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.173713 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251023 20:12:38.173959 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.164985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:55599)
1023 20:12:38.165138 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.165170 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.165682 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.165975 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.165980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.166330 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.166652 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.166656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.167357 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.167358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.167773 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.167775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.167896 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.169028 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.169059 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.170699 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.172481 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.172491 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.172494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.172840 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.173009 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.173009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.173010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.173093 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.173312 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.173317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.173467 (+ 150us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.173579 (+ 112us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.173732 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20251023 20:12:38.174438 16622 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33847
I20251023 20:12:38.174540 16622 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:38.174767 16622 heartbeater.cc:507] Master 127.9.117.254:33847 requested a full tablet report, sending...
I20251023 20:12:38.175261 16429 ts_manager.cc:194] Registered new tserver with Master: 7b5a4db60789429ba2b82ab3f1bff442 (127.9.117.193:39439)
I20251023 20:12:38.175935 16429 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:55599
WARNING: no policy specified for oryx/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:38.217527 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:33847
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:38.299131 16629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:38.299322 16629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:38.299346 16629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:38.300747 16629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:38.300794 16629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:38.300820 16629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:38.302412 16629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:38.302662 16629 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:38.302845 16629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:38.305541 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.305536 16635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.305828 16634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:38.306032 16629 server_base.cc:1047] running on GCE node
I20251023 20:12:38.306196 16629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:38.306378 16629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:38.307538 16629 hybrid_clock.cc:648] HybridClock initialized: now 1761250358307521 us; error 29 us; skew 500 ppm
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:38.310166 16629 init.cc:377] Logged in from keytab as oryx/127.9.117.194@KRBTEST.COM (short username oryx)
I20251023 20:12:38.310756 16629 webserver.cc:492] Webserver started at http://127.9.117.194:40335/ using document root <none> and password file <none>
I20251023 20:12:38.310961 16629 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:38.310998 16629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:38.311112 16629 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:38.312080 16629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "71cd94153d6a4376935a19eec2ea6b3b"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "7c5d096d8ec5e1428d4e76f0d1580b27"
server_key_iv: "270f3029949fb555a74740fb4c7efae0"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.312390 16629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "71cd94153d6a4376935a19eec2ea6b3b"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "7c5d096d8ec5e1428d4e76f0d1580b27"
server_key_iv: "270f3029949fb555a74740fb4c7efae0"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.313666 16629 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:38.314476 16644 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.314677 16629 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:38.314752 16629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "71cd94153d6a4376935a19eec2ea6b3b"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "7c5d096d8ec5e1428d4e76f0d1580b27"
server_key_iv: "270f3029949fb555a74740fb4c7efae0"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.314810 16629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:38.322917 16629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:38.324761 16629 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:38.324894 16629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:38.325088 16629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:38.325397 16629 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:38.325429 16629 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.325466 16629 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:38.325491 16629 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.331238 16629 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:40477
I20251023 20:12:38.331315 16757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:40477 every 8 connection(s)
I20251023 20:12:38.331584 16629 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:12:38.331983 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 16629
I20251023 20:12:38.332063 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:38.332226 9687 external_mini_cluster.cc:1468] Setting key 56772347a4efcb68a7645cdafb72210d
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.9.117.195@KRBTEST.COM; defaulting to no policy
I20251023 20:12:38.340582 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.332312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:40941 (local address 127.9.117.254:33847)
1023 20:12:38.332376 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.332378 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.332541 (+ 163us) server_negotiation.cc:408] Connection header received
1023 20:12:38.333169 (+ 628us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.333170 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.333186 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.333245 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.334028 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.334529 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.335086 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.335268 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.337630 (+ 2362us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.337635 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.337636 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.337642 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.339254 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.339705 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.339706 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.339706 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.339728 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.339920 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.339921 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.339922 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.340034 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.340098 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.340375 (+ 277us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.340480 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251023 20:12:38.340751 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.332270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:40941)
1023 20:12:38.332420 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.332452 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.333079 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.333308 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.333312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.333627 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.333921 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.333924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.334617 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.334618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.335003 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.335005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.335137 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.336099 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.336116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.337538 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.339352 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.339359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.339363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.339606 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.339790 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.339791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.339791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.339865 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.340093 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.340099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.340245 (+ 146us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.340349 (+ 104us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.340510 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20251023 20:12:38.341367 16758 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33847
I20251023 20:12:38.341466 16758 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:38.341707 16758 heartbeater.cc:507] Master 127.9.117.254:33847 requested a full tablet report, sending...
I20251023 20:12:38.342237 16429 ts_manager.cc:194] Registered new tserver with Master: 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.195@KRBTEST.COM" created.
I20251023 20:12:38.342712 16429 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:40941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:38.379895 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:33847
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:38.463122 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:38.463317 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:38.463341 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:38.464864 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:38.464910 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:38.464936 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:38.466509 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:38.466720 16765 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:38.466888 16765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:38.469961 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.469941 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:38.470121 16765 server_base.cc:1047] running on GCE node
W20251023 20:12:38.469945 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:38.470403 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:38.470623 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:38.471773 16765 hybrid_clock.cc:648] HybridClock initialized: now 1761250358471754 us; error 32 us; skew 500 ppm
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:38.474334 16765 init.cc:377] Logged in from keytab as oryx/127.9.117.195@KRBTEST.COM (short username oryx)
I20251023 20:12:38.474874 16765 webserver.cc:492] Webserver started at http://127.9.117.195:34635/ using document root <none> and password file <none>
I20251023 20:12:38.475060 16765 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:38.475102 16765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:38.475220 16765 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:38.476193 16765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "528e68b3955f41478e86fcd969d8e333"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "88977434670608788da67b3e8fb18978"
server_key_iv: "482ceb56c3f33411d676f87c44877460"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.476504 16765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "528e68b3955f41478e86fcd969d8e333"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "88977434670608788da67b3e8fb18978"
server_key_iv: "482ceb56c3f33411d676f87c44877460"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.477804 16765 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:12:38.478621 16780 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.478845 16765 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:38.478932 16765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "528e68b3955f41478e86fcd969d8e333"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "88977434670608788da67b3e8fb18978"
server_key_iv: "482ceb56c3f33411d676f87c44877460"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.478997 16765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:38.497877 16765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:38.499691 16765 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:38.499831 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:38.500084 16765 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:38.500412 16765 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:38.500447 16765 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.500483 16765 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:38.500509 16765 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.506675 16765 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:33119
I20251023 20:12:38.506739 16893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:33119 every 8 connection(s)
I20251023 20:12:38.507043 16765 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:38.514636 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 16765
I20251023 20:12:38.514739 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:38.514927 9687 external_mini_cluster.cc:1468] Setting key a2bd5e1e4d2c2252a78c5114a59ba352
I20251023 20:12:38.515599 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.507757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:52705 (local address 127.9.117.254:33847)
1023 20:12:38.507825 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.507828 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.507975 (+ 147us) server_negotiation.cc:408] Connection header received
1023 20:12:38.508455 (+ 480us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.508456 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.508467 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.508521 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.509344 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.509831 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.510408 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.510527 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.513003 (+ 2476us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.513007 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.513007 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.513014 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.514235 (+ 1221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.514682 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.514684 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.514684 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.514702 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.514924 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.514925 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.514926 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.515042 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.515105 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.515416 (+ 311us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.515511 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:12:38.515784 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.507718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:52705)
1023 20:12:38.507863 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.507890 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.508385 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.508587 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.508596 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.508969 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.509260 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.509263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.509945 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.509946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.510338 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.510340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.510449 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.511396 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.511412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.512922 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.514311 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.514318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.514320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.514568 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.514762 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.514765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.514766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.514859 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.515112 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.515118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.515272 (+ 154us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.515381 (+ 109us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.515527 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20251023 20:12:38.516197 16894 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33847
I20251023 20:12:38.516302 16894 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:38.516498 16894 heartbeater.cc:507] Master 127.9.117.254:33847 requested a full tablet report, sending...
I20251023 20:12:38.516862 16429 ts_manager.cc:194] Registered new tserver with Master: 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119)
I20251023 20:12:38.517234 16429 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:52705
I20251023 20:12:38.526825 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.117.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:38.572263 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.253:36315
--webserver_interface=127.9.117.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33847,127.9.117.253:36315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:38.654798 16901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:38.655002 16901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:38.655027 16901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:38.656545 16901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:38.656592 16901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:38.656610 16901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:38.656622 16901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:38.656634 16901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:38.658417 16901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33847,127.9.117.253:36315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.253:36315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.117.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
W20251023 20:12:38.658532 16901 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.117.254:33847,127.9.117.253:36315'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251023 20:12:38.658681 16901 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:38.658881 16901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:38.661459 16909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.661564 16907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.661473 16906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:38.661696 16901 server_base.cc:1047] running on GCE node
I20251023 20:12:38.661917 16901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:38.662170 16901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:38.663306 16901 hybrid_clock.cc:648] HybridClock initialized: now 1761250358663288 us; error 35 us; skew 500 ppm
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:38.666002 16901 init.cc:377] Logged in from keytab as oryx/127.9.117.253@KRBTEST.COM (short username oryx)
I20251023 20:12:38.666610 16901 webserver.cc:492] Webserver started at http://127.9.117.253:37543/ using document root <none> and password file <none>
I20251023 20:12:38.666801 16901 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:38.666848 16901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:38.666972 16901 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:38.667961 16901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data/instance:
uuid: "4ab6da4f96394fada9592954ac0c7e69"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "db1bd057b0b0fa44eb61ed10bf0225ac"
server_key_iv: "edb2408d2a39ef9ba51818ed2fba7b37"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.668277 16901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal/instance:
uuid: "4ab6da4f96394fada9592954ac0c7e69"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "db1bd057b0b0fa44eb61ed10bf0225ac"
server_key_iv: "edb2408d2a39ef9ba51818ed2fba7b37"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.669591 16901 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:38.670359 16916 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.670547 16901 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:38.670620 16901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
uuid: "4ab6da4f96394fada9592954ac0c7e69"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "db1bd057b0b0fa44eb61ed10bf0225ac"
server_key_iv: "edb2408d2a39ef9ba51818ed2fba7b37"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.670684 16901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:38.708261 16901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:38.710213 16901 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:38.710346 16901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:38.714886 16901 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.253:36315
I20251023 20:12:38.714968 16968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.253:36315 every 8 connection(s)
I20251023 20:12:38.715286 16901 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
I20251023 20:12:38.715984 16969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:38.717099 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 16901
I20251023 20:12:38.718520 16969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.117.253@KRBTEST.COM
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.253@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:38.727092 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.719294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.9.117.254:33847)
1023 20:12:38.719343 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.719345 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.719505 (+ 160us) server_negotiation.cc:408] Connection header received
1023 20:12:38.719542 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.719543 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.719555 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.719642 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.720268 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.720827 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.721465 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.721604 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.724067 (+ 2463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.724072 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.724073 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.724082 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.725791 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.726209 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.726210 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.726211 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.726238 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.726421 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.726421 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.726421 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.726574 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.726669 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.726912 (+ 243us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.727000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:12:38.727175 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.719190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42260)
1023 20:12:38.719396 (+ 206us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.719427 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.719502 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.719653 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.719654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.719973 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.720176 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.720179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.720899 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.720901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.721385 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.721387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.721469 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.721908 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.721932 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.723982 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.725875 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.725881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.725884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.726138 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.726294 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.726294 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.726295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.726361 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.726654 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.726660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.726727 (+ 67us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.726850 (+ 123us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.726987 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20251023 20:12:38.727438 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.717826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.9.117.253:36315)
1023 20:12:38.718000 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.718011 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.718032 (+ 21us) server_negotiation.cc:408] Connection header received
1023 20:12:38.718368 (+ 336us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.718373 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.718417 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.718537 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.719488 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.720330 (+ 842us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.720929 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.721112 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.723473 (+ 2361us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.723482 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.723482 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.723490 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.725838 (+ 2348us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.726215 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.726218 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.726218 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.726269 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.726420 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.726421 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.726422 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.726657 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.726774 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.727242 (+ 468us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.727336 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20251023 20:12:38.727779 16969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.253@KRBTEST.COM for oryx/127.9.117.253@KRBTEST.COM
I20251023 20:12:38.733381 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.728060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.0.0.1:40838)
1023 20:12:38.728135 (+ 75us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.728145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.728190 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.728278 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.728279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.728392 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.728532 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.728534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.729111 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.729112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.729437 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.729438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.729495 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.729896 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.729900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.731012 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.732599 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.732601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.732601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.732791 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.732935 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.732935 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.732936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.732960 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.733127 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.733127 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.733160 (+ 33us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.733214 (+ 54us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.733291 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20251023 20:12:38.733434 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.728096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.9.117.253:36315)
1023 20:12:38.728133 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.728135 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.728202 (+ 67us) server_negotiation.cc:408] Connection header received
1023 20:12:38.728232 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.728232 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.728240 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.728276 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.728585 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.729059 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.729504 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.729660 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.731092 (+ 1432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.731094 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.731095 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.731101 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.732509 (+ 1408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.732870 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.732870 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.732871 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.732888 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.733000 (+ 112us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.733001 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.733001 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.733082 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.733127 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.733235 (+ 108us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.733337 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":18}
I20251023 20:12:38.734689 16969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Bootstrap starting.
I20251023 20:12:38.735519 16969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:38.735785 16969 log.cc:826] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:38.736519 16969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: No bootstrap required, opened a new log
I20251023 20:12:38.737741 16969 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } }
I20251023 20:12:38.737864 16969 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:38.737885 16969 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab6da4f96394fada9592954ac0c7e69, State: Initialized, Role: FOLLOWER
I20251023 20:12:38.737977 16969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } }
I20251023 20:12:38.738263 16969 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:38.738221 16975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } } }
I20251023 20:12:38.738322 16975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: This master's current role is: FOLLOWER
W20251023 20:12:38.739976 16986 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:12:38.740038 16986 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:12:38.741454 16901 master_runner.cc:428] Detected that this master 4ab6da4f96394fada9592954ac0c7e69 is joining an existing cluster
I20251023 20:12:38.741503 16901 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.117.253:36315
I20251023 20:12:38.748617 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.742436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42266)
1023 20:12:38.742633 (+ 197us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.742646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.742708 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.742884 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.742885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.743149 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.743470 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.743473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.744345 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.744346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.744774 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.744776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.744848 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.745477 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.745480 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.745759 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.747701 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.747702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.747702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.747870 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.748053 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.748054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.748055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.748085 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.748322 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.748322 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.748361 (+ 39us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.748411 (+ 50us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.748502 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":114,"threads_started":1}
I20251023 20:12:38.748637 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.742676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.9.117.254:33847)
1023 20:12:38.742733 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.742735 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.742745 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:38.742773 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.742774 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.742787 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.742866 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.743585 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.744252 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.744868 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.745450 (+ 582us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.745835 (+ 385us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.745839 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.745839 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.745846 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.747579 (+ 1733us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.747942 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.747943 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.747943 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.747970 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.748147 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.748148 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.748148 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.748267 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.748335 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.748448 (+ 113us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.748540 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250357, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.254@KRBTEST.COM for oryx/127.9.117.253@KRBTEST.COM
I20251023 20:12:38.781592 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.751384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.0.0.1:40850)
1023 20:12:38.751565 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.751595 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.751668 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.755619 (+ 3951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.755631 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.756129 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.756406 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.756409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.757439 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.757440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.758108 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.758112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.758208 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.758910 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.758915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.760604 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.780312 (+ 19708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.780327 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.780332 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.780657 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.780882 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.780883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.780883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.780920 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.781190 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.781198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.781258 (+ 60us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.781344 (+ 86us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.781475 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20251023 20:12:38.781677 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.742448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.0.0.1:40840)
1023 20:12:38.742792 (+ 344us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.742800 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.742867 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.743001 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.743001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.743181 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.743871 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.743874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.744792 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.744793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.745190 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.745192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.745277 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.777630 (+ 32353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.777637 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.778302 (+ 665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.780627 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.780628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.780629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.780859 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.781022 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.781023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.781023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.781050 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.781291 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.781292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.781330 (+ 38us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.781391 (+ 61us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.781507 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":99,"threads_started":1}
I20251023 20:12:38.782044 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.751444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.9.117.253:36315)
1023 20:12:38.755411 (+ 3967us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.755413 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.755426 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:12:38.755462 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.755463 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.755474 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.755537 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.756515 (+ 978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.757335 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.758209 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.758359 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.760698 (+ 2339us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.760703 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.760704 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.760711 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.780180 (+ 19469us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.780780 (+ 600us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.780781 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.780781 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.780806 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.780982 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.780982 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.780983 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.781109 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.781159 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.781841 (+ 682us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.781946 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3935,"thread_start_us":124,"threads_started":1}
I20251023 20:12:38.782598 16429 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.9.117.253:36315: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69"
member_type: NON_VOTER
last_known_addr {
host: "127.9.117.253"
port: 36315
}
attrs {
promote: true
}
}
dest_uuid: "f53480ec5ac94aed91148cbc54a3e99f"
cas_config_opid_index: -1
I20251023 20:12:38.782822 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.742644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.9.117.253:36315)
1023 20:12:38.742683 (+ 39us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.742685 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.742849 (+ 164us) server_negotiation.cc:408] Connection header received
1023 20:12:38.742936 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.742937 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.742945 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.742988 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.743974 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.744665 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.745264 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.745398 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.778425 (+ 33027us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.778428 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.778428 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.778437 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.780505 (+ 2068us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.780936 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.780937 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.780937 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.780964 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.781104 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.781104 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.781105 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.781214 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.781267 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.782614 (+ 1347us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.782719 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:12:38.782930 16429 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } }
I20251023 20:12:38.783872 16997 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4ab6da4f96394fada9592954ac0c7e69 (127.9.117.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } }
I20251023 20:12:38.784093 16943 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:38.785106 16997 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.9.117.253:36315
I20251023 20:12:38.785265 16997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } }
I20251023 20:12:38.785322 16997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:38.785346 16901 master.cc:561] Master@127.9.117.253:36315 shutting down...
I20251023 20:12:38.785655 16943 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Refusing update from remote peer f53480ec5ac94aed91148cbc54a3e99f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251023 20:12:38.786063 16998 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:12:38.786798 16413 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251023 20:12:38.787271 16413 consensus_peers.cc:597] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f -> Peer 4ab6da4f96394fada9592954ac0c7e69 (127.9.117.253:36315): Couldn't send request to peer 4ab6da4f96394fada9592954ac0c7e69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:12:38.788985 16901 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:38.789075 16901 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:38.789119 16901 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: stopping tablet replica
WARNING: no policy specified for oryx/127.9.117.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.252@KRBTEST.COM" created.
I20251023 20:12:38.802475 16901 master.cc:583] Master@127.9.117.253:36315 shutdown complete.
I20251023 20:12:38.802531 16901 master_runner.cc:305] Clearing existing system tablet
I20251023 20:12:38.802711 16901 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:38.802772 16901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:38.803452 16901 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.804021 17004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.804162 16901 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:38.804242 16901 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:12:38.804271 16901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
uuid: "4ab6da4f96394fada9592954ac0c7e69"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "db1bd057b0b0fa44eb61ed10bf0225ac"
server_key_iv: "edb2408d2a39ef9ba51818ed2fba7b37"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.805595 16901 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:38.807332 16901 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251023 20:12:38.807408 16901 log.cc:1199] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251023 20:12:38.807650 16901 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Deleting consensus metadata
I20251023 20:12:38.807781 16901 master_runner.cc:315] Copying system tablet from 127.9.117.254:33847
I20251023 20:12:38.808497 16901 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: tablet copy: Beginning tablet copy session from remote peer at address 127.9.117.254:33847
WARNING: no policy specified for HTTP/127.9.117.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.252@KRBTEST.COM" created.
I20251023 20:12:38.819602 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.808868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42268)
1023 20:12:38.812908 (+ 4040us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:38.812924 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:38.813029 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:38.813214 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:38.813215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:38.813448 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:38.813730 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.813734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.814890 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.814891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:38.815532 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:38.815535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.815610 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.816212 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:38.816217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:38.816506 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:38.818486 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.818488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.818488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.818706 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.818938 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:38.818939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:38.818940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:38.818973 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:38.819225 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:38.819226 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:38.819279 (+ 53us) client_negotiation.cc:769] Sending connection context
1023 20:12:38.819350 (+ 71us) client_negotiation.cc:240] Negotiation successful
1023 20:12:38.819476 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4010,"thread_start_us":82,"threads_started":1}
I20251023 20:12:38.819607 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:38.808946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.9.117.254:33847)
1023 20:12:38.809000 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:38.809003 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:38.813034 (+ 4031us) server_negotiation.cc:408] Connection header received
1023 20:12:38.813093 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:38.813095 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:38.813111 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:38.813200 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:38.813902 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.814769 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:38.815636 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:38.815767 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:38.816604 (+ 837us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:38.816609 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:38.816609 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:38.816617 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:38.818364 (+ 1747us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.818832 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.818834 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.818835 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.818865 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:38.819029 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:38.819030 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:38.819030 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:38.819162 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:38.819236 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:38.819384 (+ 148us) server_negotiation.cc:300] Negotiation successful
1023 20:12:38.819481 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:12:38.819954 16449 tablet_copy_service.cc:140] P f53480ec5ac94aed91148cbc54a3e99f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4ab6da4f96394fada9592954ac0c7e69 ({username='oryx', principal='oryx/127.9.117.253@KRBTEST.COM'} at 127.0.0.1:42268)
I20251023 20:12:38.820050 16449 tablet_copy_service.cc:161] P f53480ec5ac94aed91148cbc54a3e99f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4ab6da4f96394fada9592954ac0c7e69 at {username='oryx', principal='oryx/127.9.117.253@KRBTEST.COM'} at 127.0.0.1:42268: session id = 4ab6da4f96394fada9592954ac0c7e69-00000000000000000000000000000000
I20251023 20:12:38.820886 16449 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 20:12:38.821548 16901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251023 20:12:38.823429 16901 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: tablet copy: Starting download of 0 data blocks...
I20251023 20:12:38.823593 16901 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: tablet copy: Starting download of 1 WAL segments...
I20251023 20:12:38.824658 16901 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 20:12:38.825835 16449 tablet_copy_service.cc:342] P f53480ec5ac94aed91148cbc54a3e99f: Request end of tablet copy session 4ab6da4f96394fada9592954ac0c7e69-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.117.253@KRBTEST.COM'} at 127.0.0.1:42268
I20251023 20:12:38.825924 16449 tablet_copy_service.cc:434] P f53480ec5ac94aed91148cbc54a3e99f: ending tablet copy session 4ab6da4f96394fada9592954ac0c7e69-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4ab6da4f96394fada9592954ac0c7e69
W20251023 20:12:38.827787 16901 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251023 20:12:38.828665 16901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:38.830104 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:38.830331 16901 server_base.cc:1047] running on GCE node
W20251023 20:12:38.830307 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.830431 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:38.830636 16901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:38.830791 16901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:38.831908 16901 hybrid_clock.cc:648] HybridClock initialized: now 1761250358831902 us; error 24 us; skew 500 ppm
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:38.836100 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.252:34927
--webserver_interface=127.9.117.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33847,127.9.117.253:36315,127.9.117.252:34927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:12:38.836524 16901 init.cc:377] Logged in from keytab as oryx/127.9.117.253@KRBTEST.COM (short username oryx)
I20251023 20:12:38.836890 16901 webserver.cc:492] Webserver started at http://127.9.117.253:35533/ using document root <none> and password file <none>
I20251023 20:12:38.837018 16901 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:38.837056 16901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:38.837878 16901 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:38.838409 17024 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.838531 16901 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.838567 16901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
uuid: "4ab6da4f96394fada9592954ac0c7e69"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "db1bd057b0b0fa44eb61ed10bf0225ac"
server_key_iv: "edb2408d2a39ef9ba51818ed2fba7b37"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.838632 16901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:38.844173 16901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:38.846213 16901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:38.849455 16901 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.253:36315
I20251023 20:12:38.849885 16901 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
I20251023 20:12:38.850154 17076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.253:36315 every 8 connection(s)
I20251023 20:12:38.850818 17077 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:12:38.851138 17077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Bootstrap starting.
I20251023 20:12:38.853397 17077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:12:38.853593 17077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Bootstrap complete.
I20251023 20:12:38.853850 17077 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } }
I20251023 20:12:38.853926 17077 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ab6da4f96394fada9592954ac0c7e69, State: Initialized, Role: LEARNER
I20251023 20:12:38.853968 17077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } }
I20251023 20:12:38.854117 17077 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:38.855389 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } }
I20251023 20:12:38.855489 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: This master's current role is: LEARNER
I20251023 20:12:38.856112 17091 catalog_manager.cc:1269] Loaded cluster ID: b9ef92cf951f4d268d65e7afaf5f4080
I20251023 20:12:38.856155 17091 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: loading cluster ID for follower catalog manager: success
I20251023 20:12:38.857015 17091 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: acquiring CA information for follower catalog manager: success
I20251023 20:12:38.857260 17091 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251023 20:12:38.955068 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:38.955353 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:38.955401 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:38.957564 17022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:38.957639 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:38.957659 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:38.957675 17022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:38.957705 17022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:38.960004 17022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33847,127.9.117.253:36315,127.9.117.252:34927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.252:34927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.117.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:38.960279 17022 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:38.960503 17022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:38.963141 17099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.963196 17097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:38.963341 17096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:38.963349 17022 server_base.cc:1047] running on GCE node
I20251023 20:12:38.963559 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:38.963807 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:38.964946 17022 hybrid_clock.cc:648] HybridClock initialized: now 1761250358964923 us; error 36 us; skew 500 ppm
Oct 23 20:12:38 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:38.967625 17022 init.cc:377] Logged in from keytab as oryx/127.9.117.252@KRBTEST.COM (short username oryx)
I20251023 20:12:38.968204 17022 webserver.cc:492] Webserver started at http://127.9.117.252:40157/ using document root <none> and password file <none>
I20251023 20:12:38.968389 17022 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:38.968429 17022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:38.968550 17022 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:38.969614 17022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data/instance:
uuid: "1eaa0286b9bc4f4594974bceb95a3741"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "ccf11b73f39b7d77ae5ec0e1be976c48"
server_key_iv: "6501a3255a3936008ac326746d451b34"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.969924 17022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal/instance:
uuid: "1eaa0286b9bc4f4594974bceb95a3741"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "ccf11b73f39b7d77ae5ec0e1be976c48"
server_key_iv: "6501a3255a3936008ac326746d451b34"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.971119 17022 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:38.971791 17106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:38.971951 17022 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:38.972028 17022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
uuid: "1eaa0286b9bc4f4594974bceb95a3741"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "ccf11b73f39b7d77ae5ec0e1be976c48"
server_key_iv: "6501a3255a3936008ac326746d451b34"
server_key_version: "encryptionkey@0"
I20251023 20:12:38.972102 17022 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:39.001164 17022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:39.003445 17022 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:39.003609 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:39.008196 17022 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.252:34927
I20251023 20:12:39.008230 17158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.252:34927 every 8 connection(s)
I20251023 20:12:39.008572 17022 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
I20251023 20:12:39.009253 17159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:39.011263 17159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:12:39.012151 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 17022
Oct 23 20:12:39 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.252@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
Oct 23 20:12:39 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.117.252@KRBTEST.COM
I20251023 20:12:39.020509 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.012081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.9.117.254:33847)
1023 20:12:39.012153 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.012156 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.012290 (+ 134us) server_negotiation.cc:408] Connection header received
1023 20:12:39.012905 (+ 615us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.012906 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.012916 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.012957 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.013731 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.014278 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.014988 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.015157 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.017298 (+ 2141us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.017303 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.017304 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.017313 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.019158 (+ 1845us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.019670 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.019671 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.019671 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.019690 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.019875 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.019875 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.019876 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.019971 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.020021 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.020334 (+ 313us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.020431 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251023 20:12:39.020690 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.011987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42282)
1023 20:12:39.012168 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.012200 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.012786 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.012988 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.012993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.013390 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:39.013636 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.013640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.014348 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.014349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.014921 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.014923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.015017 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.015603 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:39.015629 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:39.017183 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:39.019249 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.019255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.019258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.019595 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.019744 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.019745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.019745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.019818 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.020025 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:39.020029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:39.020206 (+ 177us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.020296 (+ 90us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.020420 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20251023 20:12:39.021304 17159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:12:39.021638 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.012681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.9.117.252:34927)
1023 20:12:39.012864 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.012876 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.012898 (+ 22us) server_negotiation.cc:408] Connection header received
1023 20:12:39.012975 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.012977 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.012994 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.013072 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.013769 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.014650 (+ 881us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.015267 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.015478 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.017899 (+ 2421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.017907 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.017908 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.017913 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.020086 (+ 2173us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.020430 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.020432 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.020433 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.020462 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.020662 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.020663 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.020664 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.020838 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.021389 (+ 551us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.021442 (+ 53us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.021533 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
Oct 23 20:12:39 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.252@KRBTEST.COM for oryx/127.9.117.253@KRBTEST.COM
I20251023 20:12:39.028628 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.021604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.0.0.1:40854)
1023 20:12:39.021653 (+ 49us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.021663 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.021703 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.022015 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.022016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.022162 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:39.022358 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.022360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.023314 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.023315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.023764 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.023767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.023835 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.024320 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:39.024325 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:39.025533 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:39.027334 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.027336 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.027337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.027490 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.027876 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.027877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.027878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.027910 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.028281 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:39.028282 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:39.028335 (+ 53us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.028408 (+ 73us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.028515 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20251023 20:12:39.028939 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.021696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.9.117.253:36315)
1023 20:12:39.021869 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.021872 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.021885 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:12:39.021912 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.021913 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.021927 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.022425 (+ 498us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.022480 (+ 55us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.023212 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.023859 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.024053 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.025624 (+ 1571us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.025629 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.025629 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.025637 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.027228 (+ 1591us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.027694 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.027695 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.027696 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.027732 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.028048 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.028049 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.028049 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.028180 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.028258 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.028529 (+ 271us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.028678 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20251023 20:12:39.029596 17159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:12:39 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250358, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.252@KRBTEST.COM for oryx/127.9.117.252@KRBTEST.COM
I20251023 20:12:39.036507 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.029931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54266)
1023 20:12:39.030044 (+ 113us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.030064 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.030129 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.030262 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.030263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.030418 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:39.030551 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.030553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.031303 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.031305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.031667 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.031669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.031737 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.032271 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:39.032275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:39.033627 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:39.035593 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.035594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.035595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.035752 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.035929 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.035930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.035930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.035951 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.036213 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:39.036214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:39.036260 (+ 46us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.036331 (+ 71us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.036428 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20251023 20:12:39.036671 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.029998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.9.117.252:34927)
1023 20:12:39.030073 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.030075 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.030111 (+ 36us) server_negotiation.cc:408] Connection header received
1023 20:12:39.030190 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.030191 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.030203 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.030258 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.030624 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.031207 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.031739 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.031940 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.033791 (+ 1851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.033795 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.033795 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.033802 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.035465 (+ 1663us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.035838 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.035840 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.035841 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.035865 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.035993 (+ 128us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.035993 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.035994 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.036098 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.036153 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.036429 (+ 276us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.036511 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251023 20:12:39.038198 17159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Bootstrap starting.
I20251023 20:12:39.039093 17159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:39.039431 17159 log.cc:826] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:39.040452 17159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: No bootstrap required, opened a new log
I20251023 20:12:39.041877 17159 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } }
I20251023 20:12:39.042012 17159 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:39.042033 17159 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaa0286b9bc4f4594974bceb95a3741, State: Initialized, Role: FOLLOWER
I20251023 20:12:39.042125 17159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } }
I20251023 20:12:39.042428 17159 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:39.042431 17165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } } }
I20251023 20:12:39.042572 17165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: This master's current role is: FOLLOWER
W20251023 20:12:39.044302 17176 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:12:39.044370 17176 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:12:39.046429 17022 master_runner.cc:428] Detected that this master 1eaa0286b9bc4f4594974bceb95a3741 is joining an existing cluster
I20251023 20:12:39.046476 17022 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.117.252:34927
I20251023 20:12:39.078784 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.073632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42304)
1023 20:12:39.073716 (+ 84us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.073728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.073765 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.074134 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.074134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.074136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:39.074347 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.074349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.075454 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.075455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.077687 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.077691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.078258 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.078322 (+ 64us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.078363 (+ 41us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.078416 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20251023 20:12:39.080201 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.047695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.9.117.253:36315)
1023 20:12:39.047743 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.047745 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.048161 (+ 416us) server_negotiation.cc:408] Connection header received
1023 20:12:39.048198 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.048199 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.048220 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.048283 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.073585 (+ 25302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.074397 (+ 812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.075282 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.075474 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.076772 (+ 1298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.076777 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.076777 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.076786 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.078929 (+ 2143us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.079343 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.079344 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.079345 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.079373 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.079550 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.079551 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.079551 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.079680 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.079724 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.079963 (+ 239us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.080069 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:12:39.080585 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.047504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.0.0.1:40860)
1023 20:12:39.048083 (+ 579us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.048091 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.048148 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.048297 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.048298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.073055 (+ 24757us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:39.073441 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.073445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.074498 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.074498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.075190 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.075193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.075532 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.076227 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:39.076231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:39.076648 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:39.079025 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.079026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.079027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.079233 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.079446 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.079447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.079448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.079477 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.079770 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:39.079771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:39.079829 (+ 58us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.080353 (+ 524us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.080469 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":158,"threads_started":1}
I20251023 20:12:39.081023 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.073832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.9.117.254:33847)
1023 20:12:39.073992 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.073994 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.074005 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:12:39.074046 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.074046 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.074057 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.074404 (+ 347us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:39.074441 (+ 37us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.075362 (+ 921us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.079761 (+ 4399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.080487 (+ 726us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.080527 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.080554 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.080590 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
I20251023 20:12:39.082304 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.048010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.9.117.252:34927)
1023 20:12:39.048052 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.048054 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.048284 (+ 230us) server_negotiation.cc:408] Connection header received
1023 20:12:39.048357 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.048358 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.048368 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.048419 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.073651 (+ 25232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.074316 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.075800 (+ 1484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.075974 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.077705 (+ 1731us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.077709 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.077709 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.077716 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.080117 (+ 2401us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.081619 (+ 1502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.081620 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.081621 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.081643 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.081799 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.081800 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.081801 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.081891 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.081942 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.082122 (+ 180us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.082216 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251023 20:12:39.082304 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.047555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.9.117.254:33847)
1023 20:12:39.047639 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.047641 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.047798 (+ 157us) server_negotiation.cc:408] Connection header received
1023 20:12:39.047879 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.047880 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.047893 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.047959 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.075241 (+ 27282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.075957 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.076753 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.076922 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.078283 (+ 1361us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.078287 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.078288 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.078295 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.081006 (+ 2711us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.081577 (+ 571us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.081578 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.081578 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.081606 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.081786 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.081787 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.081787 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.081896 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.081940 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.082136 (+ 196us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.082220 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251023 20:12:39.082626 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.047874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54268)
1023 20:12:39.048228 (+ 354us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.048236 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.048307 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.048429 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.048430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.073321 (+ 24891us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:39.073530 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.073532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.074659 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.074660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.075130 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.075133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.076040 (+ 907us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.077095 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:39.077099 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:39.077598 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:39.081103 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.081104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.081104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.081294 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.081708 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.081708 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.081709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.081737 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.081970 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:39.081971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:39.082005 (+ 34us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.082420 (+ 415us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.082524 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":60,"threads_started":1}
I20251023 20:12:39.082881 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.047484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42292)
1023 20:12:39.047729 (+ 245us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.047742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.047812 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.047965 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.047966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.073054 (+ 25088us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:39.074546 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.074548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.076050 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.076051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.076611 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.076615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.076978 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.077796 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:39.077801 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:39.078177 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:39.081356 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.081357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.081357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.081500 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.081681 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.081681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.081682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.081713 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.081987 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:39.081988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:39.082021 (+ 33us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.082421 (+ 400us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.082776 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":221,"thread_start_us":270,"threads_started":1}
I20251023 20:12:39.085039 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.081860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.0.0.1:40864)
1023 20:12:39.082005 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.082015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.082056 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.082598 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.082599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.082601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:39.082796 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.082798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.083982 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.083983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.084821 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.084825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.084901 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.084905 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.084940 (+ 35us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.084978 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20251023 20:12:39.085659 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.081962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.9.117.253:36315)
1023 20:12:39.082131 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.082133 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.082141 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:12:39.082187 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.082188 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.082221 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.082281 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:39.083193 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.083910 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.084908 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.085390 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.085513 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.085537 (+ 24us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.085582 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
Oct 23 20:12:39 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250357, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.254@KRBTEST.COM for oryx/127.9.117.252@KRBTEST.COM
I20251023 20:12:39.092470 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.085235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54284)
1023 20:12:39.085349 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.085360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.085396 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.085569 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.085570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.085769 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:39.085919 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.085922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.086667 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.086669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.087186 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.087188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.087274 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.087775 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:39.087779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:39.089404 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:39.091169 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.091171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.091172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.091356 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.091705 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.091706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.091706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.091750 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.092111 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:39.092112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:39.092166 (+ 54us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.092242 (+ 76us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.092368 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20251023 20:12:39.092511 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.085332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.9.117.252:34927)
1023 20:12:39.085386 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.085388 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.085398 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:39.085472 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.085473 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.085485 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.085559 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.086017 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.086582 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.087267 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.087370 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.089489 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.089494 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.089494 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.089500 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.091090 (+ 1590us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.091442 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.091443 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.091443 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.091467 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.091909 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.091910 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.091910 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.092043 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.092123 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.092290 (+ 167us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.092400 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251023 20:12:39.093436 16428 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.9.117.252:34927: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741"
member_type: NON_VOTER
last_known_addr {
host: "127.9.117.252"
port: 34927
}
attrs {
promote: true
}
}
dest_uuid: "f53480ec5ac94aed91148cbc54a3e99f"
cas_config_opid_index: 5
I20251023 20:12:39.093706 16428 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } }
I20251023 20:12:39.094132 16999 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.094306 17051 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 LEARNER]: Refusing update from remote peer f53480ec5ac94aed91148cbc54a3e99f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251023 20:12:39.094544 16998 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251023 20:12:39.095155 17051 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.095276 16999 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.9.117.252:34927
I20251023 20:12:39.095355 16998 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.095425 16998 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:39.095553 17133 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:39.096518 17133 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Refusing update from remote peer f53480ec5ac94aed91148cbc54a3e99f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251023 20:12:39.096715 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f53480ec5ac94aed91148cbc54a3e99f. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.096848 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: This master's current role is: LEARNER
I20251023 20:12:39.096895 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: NON_VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: true } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.096923 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: This master's current role is: LEARNER
I20251023 20:12:39.096971 17022 master.cc:561] Master@127.9.117.252:34927 shutting down...
I20251023 20:12:39.097651 16998 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:39.098019 16998 raft_consensus.cc:1064] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: attempting to promote NON_VOTER 4ab6da4f96394fada9592954ac0c7e69 to VOTER
I20251023 20:12:39.098194 16998 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } }
I20251023 20:12:39.098706 17051 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 LEARNER]: Refusing update from remote peer f53480ec5ac94aed91148cbc54a3e99f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251023 20:12:39.099134 16997 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251023 20:12:39.099999 16997 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4ab6da4f96394fada9592954ac0c7e69 (127.9.117.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.100901 16997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
W20251023 20:12:39.100957 16414 consensus_peers.cc:597] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f -> Peer 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252:34927): Couldn't send request to peer 1eaa0286b9bc4f4594974bceb95a3741. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:12:39.100996 16997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:39.101063 16997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.101119 16997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:39.101363 17051 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4ab6da4f96394fada9592954ac0c7e69 (127.9.117.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.101516 17022 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:39.101611 17022 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:12:39.101641 16428 master.cc:679] Remote error: unable to get registration information for peer 1eaa0286b9bc4f4594974bceb95a3741: Service unavailable: service kudu.master.MasterService not registered on Master
I20251023 20:12:39.101663 17022 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: stopping tablet replica
I20251023 20:12:39.102504 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.102591 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:12:39.112323 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.112145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54288)
1023 20:12:39.112203 (+ 58us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.112273 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20251023 20:12:39.112473 16428 master.cc:679] Network error: unable to get registration information for peer 1eaa0286b9bc4f4594974bceb95a3741: Client connection negotiation failed: client connection to 127.9.117.252:34927: connect: Connection refused (error 111)
I20251023 20:12:39.114662 17022 master.cc:583] Master@127.9.117.252:34927 shutdown complete.
I20251023 20:12:39.114723 17022 master_runner.cc:305] Clearing existing system tablet
I20251023 20:12:39.114883 17022 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:39.114935 17022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:39.115695 17022 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:39.116254 17189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:39.116389 17022 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:39.116449 17022 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:39.116474 17022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
uuid: "1eaa0286b9bc4f4594974bceb95a3741"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "ccf11b73f39b7d77ae5ec0e1be976c48"
server_key_iv: "6501a3255a3936008ac326746d451b34"
server_key_version: "encryptionkey@0"
I20251023 20:12:39.117317 17022 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:39.119004 17022 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251023 20:12:39.119076 17022 log.cc:1199] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251023 20:12:39.119279 17022 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Deleting consensus metadata
I20251023 20:12:39.119402 17022 master_runner.cc:315] Copying system tablet from 127.9.117.254:33847
I20251023 20:12:39.120115 17022 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: tablet copy: Beginning tablet copy session from remote peer at address 127.9.117.254:33847
I20251023 20:12:39.125909 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.120421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42308)
1023 20:12:39.120549 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.120559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.120623 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.120830 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.120830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.121006 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:39.121196 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.121198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.122151 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.122152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.122539 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.122540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.122592 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.123098 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:39.123103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:39.123381 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:39.125102 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.125103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.125103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.125250 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.125397 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:39.125397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:39.125398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:39.125421 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:39.125631 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:39.125632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:39.125672 (+ 40us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.125721 (+ 49us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.125808 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20251023 20:12:39.125942 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.120502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.9.117.254:33847)
1023 20:12:39.120556 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.120559 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.120609 (+ 50us) server_negotiation.cc:408] Connection header received
1023 20:12:39.120673 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.120675 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.120686 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.120824 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:39.121273 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.122067 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.122605 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.122795 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.123458 (+ 663us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:39.123462 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:39.123463 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:39.123469 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:39.125008 (+ 1539us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.125311 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.125312 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.125313 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.125338 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:39.125466 (+ 128us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:39.125466 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:39.125466 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:39.125580 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:39.125646 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.125756 (+ 110us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.125848 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:12:39.126363 16449 tablet_copy_service.cc:140] P f53480ec5ac94aed91148cbc54a3e99f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1eaa0286b9bc4f4594974bceb95a3741 ({username='oryx', principal='oryx/127.9.117.252@KRBTEST.COM'} at 127.0.0.1:42308)
I20251023 20:12:39.126423 16449 tablet_copy_service.cc:161] P f53480ec5ac94aed91148cbc54a3e99f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1eaa0286b9bc4f4594974bceb95a3741 at {username='oryx', principal='oryx/127.9.117.252@KRBTEST.COM'} at 127.0.0.1:42308: session id = 1eaa0286b9bc4f4594974bceb95a3741-00000000000000000000000000000000
I20251023 20:12:39.126919 16449 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 20:12:39.127462 17022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251023 20:12:39.128997 17022 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: tablet copy: Starting download of 0 data blocks...
I20251023 20:12:39.129145 17022 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: tablet copy: Starting download of 1 WAL segments...
I20251023 20:12:39.129953 17022 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 20:12:39.130300 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.130169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54300)
1023 20:12:39.130225 (+ 56us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.130263 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20251023 20:12:39.130496 16428 master.cc:679] Network error: unable to get registration information for peer 1eaa0286b9bc4f4594974bceb95a3741: Client connection negotiation failed: client connection to 127.9.117.252:34927: connect: Connection refused (error 111)
I20251023 20:12:39.131145 16449 tablet_copy_service.cc:342] P f53480ec5ac94aed91148cbc54a3e99f: Request end of tablet copy session 1eaa0286b9bc4f4594974bceb95a3741-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.117.252@KRBTEST.COM'} at 127.0.0.1:42308
I20251023 20:12:39.131206 16449 tablet_copy_service.cc:434] P f53480ec5ac94aed91148cbc54a3e99f: ending tablet copy session 1eaa0286b9bc4f4594974bceb95a3741-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1eaa0286b9bc4f4594974bceb95a3741
W20251023 20:12:39.133003 17022 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251023 20:12:39.133960 17022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:39.135450 17197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:39.135520 17199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:39.135476 17196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:39.135735 17022 server_base.cc:1047] running on GCE node
I20251023 20:12:39.135941 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:39.136086 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:39.137203 17022 hybrid_clock.cc:648] HybridClock initialized: now 1761250359137199 us; error 32 us; skew 500 ppm
Oct 23 20:12:39 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250359, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:39.138804 17022 init.cc:377] Logged in from keytab as oryx/127.9.117.252@KRBTEST.COM (short username oryx)
I20251023 20:12:39.139086 17022 webserver.cc:492] Webserver started at http://127.9.117.252:43241/ using document root <none> and password file <none>
I20251023 20:12:39.139204 17022 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:39.139240 17022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:39.140098 17022 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:39.140815 17206 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:39.140954 17022 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:39.141009 17022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
uuid: "1eaa0286b9bc4f4594974bceb95a3741"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "ccf11b73f39b7d77ae5ec0e1be976c48"
server_key_iv: "6501a3255a3936008ac326746d451b34"
server_key_version: "encryptionkey@0"
I20251023 20:12:39.141073 17022 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:39.164484 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.164367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54310)
1023 20:12:39.164414 (+ 47us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.164444 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20251023 20:12:39.164603 16428 master.cc:679] Network error: unable to get registration information for peer 1eaa0286b9bc4f4594974bceb95a3741: Client connection negotiation failed: client connection to 127.9.117.252:34927: connect: Connection refused (error 111)
I20251023 20:12:39.173907 17022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:39.175715 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:39.176826 16622 heartbeater.cc:499] Master 127.9.117.254:33847 was elected leader, sending a full tablet report...
I20251023 20:12:39.179188 17022 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.252:34927
I20251023 20:12:39.179219 17258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.252:34927 every 8 connection(s)
I20251023 20:12:39.179551 17022 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
I20251023 20:12:39.180426 17259 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:12:39.180898 17259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Bootstrap starting.
I20251023 20:12:39.183554 17259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:12:39.183768 17259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: Bootstrap complete.
I20251023 20:12:39.183984 17259 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } }
I20251023 20:12:39.184060 17259 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1eaa0286b9bc4f4594974bceb95a3741, State: Initialized, Role: LEARNER
I20251023 20:12:39.184104 17259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } }
I20251023 20:12:39.184286 17259 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:39.184307 17262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.184424 17262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: This master's current role is: LEARNER
I20251023 20:12:39.185803 17273 catalog_manager.cc:1269] Loaded cluster ID: b9ef92cf951f4d268d65e7afaf5f4080
I20251023 20:12:39.185880 17273 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: loading cluster ID for follower catalog manager: success
I20251023 20:12:39.187239 17273 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: acquiring CA information for follower catalog manager: success
I20251023 20:12:39.187577 17273 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:12:39.233201 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.230468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54312)
1023 20:12:39.230533 (+ 65us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:39.230546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:39.230595 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:39.230851 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:39.230852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:39.230853 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:39.231061 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.231063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.231915 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.231917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:39.232919 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:39.232923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.233001 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.233007 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:12:39.233045 (+ 38us) client_negotiation.cc:240] Negotiation successful
1023 20:12:39.233100 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251023 20:12:39.233559 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:39.230550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.9.117.252:34927)
1023 20:12:39.230705 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:39.230708 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:39.230723 (+ 15us) server_negotiation.cc:408] Connection header received
1023 20:12:39.230748 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:39.230749 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:39.230772 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:39.230834 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:39.231139 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.231828 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:39.233001 (+ 1173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:39.233339 (+ 338us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:39.233428 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:39.233453 (+ 25us) server_negotiation.cc:300] Negotiation successful
1023 20:12:39.233487 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20251023 20:12:39.343521 16758 heartbeater.cc:499] Master 127.9.117.254:33847 was elected leader, sending a full tablet report...
I20251023 20:12:39.511763 17233 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251023 20:12:39.511971 17262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f53480ec5ac94aed91148cbc54a3e99f. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:39.512075 17262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: This master's current role is: LEARNER
I20251023 20:12:39.517967 16894 heartbeater.cc:499] Master 127.9.117.254:33847 was elected leader, sending a full tablet report...
I20251023 20:12:40.130972 17275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: NON_VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: true } } }
I20251023 20:12:40.131134 17275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:40.131515 17275 raft_consensus.cc:1064] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: attempting to promote NON_VOTER 1eaa0286b9bc4f4594974bceb95a3741 to VOTER
I20251023 20:12:40.131680 17275 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } }
I20251023 20:12:40.132249 17233 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 LEARNER]: Refusing update from remote peer f53480ec5ac94aed91148cbc54a3e99f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251023 20:12:40.132268 17051 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Refusing update from remote peer f53480ec5ac94aed91148cbc54a3e99f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251023 20:12:40.132483 17276 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251023 20:12:40.132652 17275 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251023 20:12:40.133386 17277 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:40.133550 17051 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:40.134488 17277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:40.134578 17277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:40.134531 17275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:40.134634 17275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:40.134830 17282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:40.134933 17282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:12:40.135183 17275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:40.135246 17275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:40.135340 17233 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:40.136544 17283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f53480ec5ac94aed91148cbc54a3e99f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:40.136651 17283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:12:40.143281 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.137437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.9.117.253:36315)
1023 20:12:40.137528 (+ 91us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.137530 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.137588 (+ 58us) server_negotiation.cc:408] Connection header received
1023 20:12:40.137651 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.137651 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.137662 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.137716 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.138447 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.139002 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.139495 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.139640 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.140814 (+ 1174us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.140819 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.140819 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.140826 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.142456 (+ 1630us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.142771 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.142772 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.142773 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.142790 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.142905 (+ 115us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.142906 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.142906 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.142992 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.143066 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.143141 (+ 75us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.143212 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":42,"threads_started":1}
I20251023 20:12:40.145742 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.143690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42324)
1023 20:12:40.143815 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.143825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.143880 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.144032 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.144033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.144035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.144209 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.144211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.144941 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.144942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.145512 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.145514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.145585 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.145590 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.145622 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.145659 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20251023 20:12:40.146174 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.143771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.9.117.254:33847)
1023 20:12:40.143928 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.143931 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.143941 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:40.143961 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.143962 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.143971 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.144026 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.144295 (+ 269us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.144861 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.145615 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.146061 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.146070 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.146092 (+ 22us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.146123 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20251023 20:12:40.152192 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.149680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.0.0.1:40882)
1023 20:12:40.149906 (+ 226us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.149917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.150070 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.150197 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.150198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.150201 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.150394 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.150396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.151117 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.151118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.151953 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.151956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.152030 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.152036 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.152068 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.152107 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20251023 20:12:40.152637 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.149736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.9.117.253:36315)
1023 20:12:40.149787 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.149789 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.150048 (+ 259us) server_negotiation.cc:408] Connection header received
1023 20:12:40.150129 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.150130 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.150141 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.150196 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.150446 (+ 250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.151025 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.152026 (+ 1001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.152494 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.152504 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.152529 (+ 25us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.152563 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251023 20:12:40.155488 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.153477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54316)
1023 20:12:40.153519 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.153530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.153568 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.153954 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.153955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.153957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.154126 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.154128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.154782 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.154783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.155337 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.155339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.155397 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.155400 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.155422 (+ 22us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.155445 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20251023 20:12:40.156021 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.153589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.9.117.252:34927)
1023 20:12:40.153803 (+ 214us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.153805 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.153815 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:40.153848 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.153849 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.153874 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.153949 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.154195 (+ 246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.154686 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.155434 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.155912 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.155919 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.155942 (+ 23us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.155969 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":154,"threads_started":1}
I20251023 20:12:40.162667 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.156999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.9.117.252:34927)
1023 20:12:40.157047 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.157048 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.157056 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:12:40.157088 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.157089 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.157096 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.157148 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.157766 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.158215 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.158853 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.158986 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.160023 (+ 1037us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.160027 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.160027 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.160034 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.161821 (+ 1787us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.162129 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.162130 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.162130 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.162152 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.162262 (+ 110us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.162263 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.162263 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.162345 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.162422 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.162509 (+ 87us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.162588 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:12:40.165380 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.163143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42332)
1023 20:12:40.163286 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.163299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.163376 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.163501 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.163501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.163504 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.163683 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.163685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.164533 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.164534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.165160 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.165163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.165235 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.165241 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.165275 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.165312 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20251023 20:12:40.165807 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.163238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.9.117.254:33847)
1023 20:12:40.163317 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.163319 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.163361 (+ 42us) server_negotiation.cc:408] Connection header received
1023 20:12:40.163432 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.163433 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.163445 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.163494 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.163772 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.164455 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.165252 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.165701 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.165710 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.165731 (+ 21us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.165757 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251023 20:12:40.168258 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.166449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.0.0.1:40886)
1023 20:12:40.166516 (+ 67us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.166528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.166571 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.166750 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.166750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.166755 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.166883 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.166884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.167516 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.167516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.168095 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.168096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.168156 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.168159 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.168183 (+ 24us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.168210 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20251023 20:12:40.168641 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.166594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.9.117.253:36315)
1023 20:12:40.166650 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.166652 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.166660 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:12:40.166682 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.166683 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.166691 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.166738 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.166968 (+ 230us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.167442 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.168176 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.168539 (+ 363us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.168546 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.168563 (+ 17us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.168588 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251023 20:12:40.170914 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.169345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.0.0.1:54344)
1023 20:12:40.169389 (+ 44us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.169402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.169433 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.169540 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.169540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.169542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.169655 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.169655 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.170215 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.170215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.170740 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.170742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.170796 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.170799 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.170833 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.170864 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20251023 20:12:40.171299 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.169405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.9.117.252:34927)
1023 20:12:40.169452 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.169454 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.169460 (+ 6us) server_negotiation.cc:408] Connection header received
1023 20:12:40.169477 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.169478 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.169485 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.169533 (+ 48us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.169708 (+ 175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.170161 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.170808 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.171214 (+ 406us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.171221 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.171238 (+ 17us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.171262 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251023 20:12:40.172099 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 16493
I20251023 20:12:40.180435 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:39439
--local_ip_for_outbound_sockets=127.9.117.193
--tserver_master_addrs=127.9.117.254:33847
--webserver_port=40521
--webserver_interface=127.9.117.193
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315,127.9.117.252:34927 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:40.266217 17290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:40.266395 17290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:40.266422 17290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:40.267885 17290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:40.267933 17290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:40.267962 17290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:40.269644 17290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:39439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=40521
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315,127.9.117.252:34927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:40.269877 17290 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:40.270079 17290 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:40.272702 17295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:40.272711 17298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:40.272924 17290 server_base.cc:1047] running on GCE node
W20251023 20:12:40.272711 17296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:40.273188 17290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:40.273402 17290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:40.274565 17290 hybrid_clock.cc:648] HybridClock initialized: now 1761250360274538 us; error 44 us; skew 500 ppm
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:40.277124 17290 init.cc:377] Logged in from keytab as oryx/127.9.117.193@KRBTEST.COM (short username oryx)
I20251023 20:12:40.277662 17290 webserver.cc:492] Webserver started at http://127.9.117.193:40521/ using document root <none> and password file <none>
I20251023 20:12:40.277844 17290 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:40.277889 17290 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:40.279150 17290 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:40.279693 17305 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.279917 17290 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:40.279989 17290 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "7b5a4db60789429ba2b82ab3f1bff442"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "2f9a3941f72dc662c782cfdf87f8b4ec"
server_key_iv: "ed62ed383f2bb0706eac094c63cd9da1"
server_key_version: "encryptionkey@0"
I20251023 20:12:40.280270 17290 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:40.293743 17290 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:40.295801 17290 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:40.295933 17290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:40.296149 17290 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:40.296447 17290 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:40.296478 17290 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.296501 17290 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:40.296530 17290 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.302366 17290 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:39439
I20251023 20:12:40.302454 17418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:39439 every 8 connection(s)
I20251023 20:12:40.302767 17290 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:40.305274 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 17290
I20251023 20:12:40.305437 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 16629
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.253@KRBTEST.COM
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.252@KRBTEST.COM
I20251023 20:12:40.313205 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:40477
--local_ip_for_outbound_sockets=127.9.117.194
--tserver_master_addrs=127.9.117.254:33847
--webserver_port=40335
--webserver_interface=127.9.117.194
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315,127.9.117.252:34927 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:40.314769 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.303469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.9.117.193:57539)
1023 20:12:40.303767 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.303777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.304279 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.304518 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.304523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.304995 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.305511 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.305516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.306849 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.306850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.307492 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.307495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.307581 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.308139 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.308168 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.310659 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.312970 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.312978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.312982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.313305 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.313503 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.313504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.313505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.313598 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.313924 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.313930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.314190 (+ 260us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.314311 (+ 121us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.314482 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20251023 20:12:40.315188 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.303866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:48525 (local address 127.9.117.252:34927)
1023 20:12:40.303933 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.303935 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.303944 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:40.304382 (+ 438us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.304383 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.304395 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.304429 (+ 34us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.308223 (+ 3794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.309073 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.309879 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.310091 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.311933 (+ 1842us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.311938 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.311938 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.311948 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.313837 (+ 1889us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.314298 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.314300 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.314300 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.314327 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.314548 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.314549 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.314550 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.314676 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.314729 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.314974 (+ 245us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.315096 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251023 20:12:40.315457 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.303520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:57539 (local address 127.9.117.253:36315)
1023 20:12:40.303700 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.303703 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.303825 (+ 122us) server_negotiation.cc:408] Connection header received
1023 20:12:40.304379 (+ 554us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.304381 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.304395 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.304454 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.305931 (+ 1477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.306723 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.307586 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.307811 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.310764 (+ 2953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.310769 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.310769 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.310779 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.312870 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.313397 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.313399 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.313399 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.313431 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.313680 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.313681 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.313682 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.313830 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.313896 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.315103 (+ 1207us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.315271 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20251023 20:12:40.315531 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.303747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.9.117.193:48525)
1023 20:12:40.303894 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.303902 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.304281 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.304519 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.304526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.305034 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.308121 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.308123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.309161 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.309162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.309747 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.309750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.309856 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.310320 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.310325 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.311801 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.313947 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.313948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.313949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.314210 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.314395 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.314396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.314397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.314426 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.314786 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.314787 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.314837 (+ 50us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.315307 (+ 470us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.315432 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20251023 20:12:40.316037 17421 heartbeater.cc:344] Connected to a master server at 127.9.117.253:36315
I20251023 20:12:40.316154 17421 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.316422 17421 heartbeater.cc:507] Master 127.9.117.253:36315 requested a full tablet report, sending...
I20251023 20:12:40.316040 17420 heartbeater.cc:344] Connected to a master server at 127.9.117.252:34927
I20251023 20:12:40.316555 17420 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.316637 17420 heartbeater.cc:507] Master 127.9.117.252:34927 requested a full tablet report, sending...
I20251023 20:12:40.316996 17041 ts_manager.cc:194] Registered new tserver with Master: 7b5a4db60789429ba2b82ab3f1bff442 (127.9.117.193:39439)
I20251023 20:12:40.317037 17222 ts_manager.cc:194] Registered new tserver with Master: 7b5a4db60789429ba2b82ab3f1bff442 (127.9.117.193:39439)
I20251023 20:12:40.317674 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.303469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:51985)
1023 20:12:40.303635 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.303666 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.304270 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.304524 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.304525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.305032 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.305676 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.305679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.306718 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.306719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.307188 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.307190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.307329 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.308247 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.308252 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.314100 (+ 5848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.316238 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.316239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.316239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.316408 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.316616 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.316617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.316618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.316651 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.317442 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.317442 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.317475 (+ 33us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.317520 (+ 45us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.317607 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20251023 20:12:40.317754 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.303558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:51985 (local address 127.9.117.254:33847)
1023 20:12:40.303611 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.303613 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.303743 (+ 130us) server_negotiation.cc:408] Connection header received
1023 20:12:40.304450 (+ 707us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.304452 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.304465 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.304514 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.305781 (+ 1267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.306608 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.307278 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.307466 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.314212 (+ 6746us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.314217 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.314218 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.314226 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.316139 (+ 1913us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.316485 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.316486 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.316487 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.316517 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.317235 (+ 718us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.317236 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.317237 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.317349 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.317402 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.317594 (+ 192us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.317690 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251023 20:12:40.318094 17419 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33847
I20251023 20:12:40.318142 17419 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.318231 17419 heartbeater.cc:507] Master 127.9.117.254:33847 requested a full tablet report, sending...
I20251023 20:12:40.318488 16428 ts_manager.cc:194] Re-registered known tserver with Master: 7b5a4db60789429ba2b82ab3f1bff442 (127.9.117.193:39439)
I20251023 20:12:40.318882 16428 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:51985
W20251023 20:12:40.398372 17426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:40.398561 17426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:40.398599 17426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:40.400030 17426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:40.400072 17426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:40.400111 17426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:40.401787 17426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:40477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=40335
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315,127.9.117.252:34927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:40.402035 17426 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:40.402227 17426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:40.404989 17432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:40.404994 17431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:40.405004 17434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:40.405203 17426 server_base.cc:1047] running on GCE node
I20251023 20:12:40.405368 17426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:40.405555 17426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:40.406716 17426 hybrid_clock.cc:648] HybridClock initialized: now 1761250360406703 us; error 24 us; skew 500 ppm
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:40.409210 17426 init.cc:377] Logged in from keytab as oryx/127.9.117.194@KRBTEST.COM (short username oryx)
I20251023 20:12:40.409744 17426 webserver.cc:492] Webserver started at http://127.9.117.194:40335/ using document root <none> and password file <none>
I20251023 20:12:40.409931 17426 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:40.409986 17426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:40.411257 17426 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:40.411758 17441 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.411933 17426 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.411998 17426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "71cd94153d6a4376935a19eec2ea6b3b"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "7c5d096d8ec5e1428d4e76f0d1580b27"
server_key_iv: "270f3029949fb555a74740fb4c7efae0"
server_key_version: "encryptionkey@0"
I20251023 20:12:40.412277 17426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:40.422330 17426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:40.424240 17426 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:40.424386 17426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:40.424621 17426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:40.424986 17426 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:40.425021 17426 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.425055 17426 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:40.425084 17426 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.431031 17426 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:40477
I20251023 20:12:40.431087 17554 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:40477 every 8 connection(s)
I20251023 20:12:40.431408 17426 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.253@KRBTEST.COM
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.252@KRBTEST.COM
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:40.441090 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 17426
I20251023 20:12:40.441260 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 16765
I20251023 20:12:40.442276 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.432301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:34193 (local address 127.9.117.253:36315)
1023 20:12:40.432361 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.432363 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.432603 (+ 240us) server_negotiation.cc:408] Connection header received
1023 20:12:40.433331 (+ 728us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.433333 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.433346 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.433380 (+ 34us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.434388 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.435196 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.435870 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.436041 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.438551 (+ 2510us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.438555 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.438556 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.438564 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.440555 (+ 1991us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441134 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.441136 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.441136 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.441163 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441393 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.441394 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.441394 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.441542 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.441614 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.442068 (+ 454us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.442179 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:12:40.442319 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.432572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:53977 (local address 127.9.117.252:34927)
1023 20:12:40.432618 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.432620 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.432909 (+ 289us) server_negotiation.cc:408] Connection header received
1023 20:12:40.433298 (+ 389us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.433300 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.433314 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.433379 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.434606 (+ 1227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.435089 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.435885 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.436062 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.438891 (+ 2829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.438896 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.438897 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.438908 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.440957 (+ 2049us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441473 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.441475 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.441475 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.441502 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441667 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.441668 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.441668 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.441801 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.441845 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.442061 (+ 216us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.442181 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:12:40.442412 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.432388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:42901 (local address 127.9.117.254:33847)
1023 20:12:40.432451 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.432453 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.432626 (+ 173us) server_negotiation.cc:408] Connection header received
1023 20:12:40.433355 (+ 729us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.433357 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.433374 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.433423 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.434452 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.435197 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.436696 (+ 1499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.437255 (+ 559us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.439431 (+ 2176us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.439436 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.439436 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.439444 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.441313 (+ 1869us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441630 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.441631 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.441631 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.441656 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441830 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.441831 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.441831 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.441947 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.441992 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.442245 (+ 253us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.442333 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251023 20:12:40.442806 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.432117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:42901)
1023 20:12:40.432555 (+ 438us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.432567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.433179 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.433478 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.433479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.433930 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.434315 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.434318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.436181 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.436181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.436606 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.436609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.436683 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.437158 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.437162 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.439313 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.441392 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.441393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.441548 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.441742 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.441742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.441769 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.442041 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.442042 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.442122 (+ 80us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.442531 (+ 409us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.442611 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20251023 20:12:40.442806 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.432117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.9.117.194:34193)
1023 20:12:40.432445 (+ 328us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.432481 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.433203 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.433469 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.433470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.433929 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.434286 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.434289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.435272 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.435272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.435790 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.435792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.435924 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.436657 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.436662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.438431 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.440682 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.440690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.440693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.441003 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.441234 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.441235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.441317 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.441701 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.441707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.441903 (+ 196us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.442378 (+ 475us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.442540 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
I20251023 20:12:40.443691 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.432426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.9.117.194:53977)
1023 20:12:40.432823 (+ 397us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.432834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.433196 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.433450 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.433455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.433934 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.434527 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.434529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.435169 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.435170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.435794 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.435795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.435940 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.436562 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.436588 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.438799 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.441053 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.441054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.441390 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.441566 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.441566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.441567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.441596 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.441893 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.441893 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.441934 (+ 41us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.443412 (+ 1478us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.443537 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":59,"threads_started":1}
I20251023 20:12:40.444124 17556 heartbeater.cc:344] Connected to a master server at 127.9.117.252:34927
I20251023 20:12:40.444231 17556 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.444451 17556 heartbeater.cc:507] Master 127.9.117.252:34927 requested a full tablet report, sending...
I20251023 20:12:40.444975 17222 ts_manager.cc:194] Registered new tserver with Master: 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477)
I20251023 20:12:40.447366 17555 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33847
I20251023 20:12:40.447409 17557 heartbeater.cc:344] Connected to a master server at 127.9.117.253:36315
I20251023 20:12:40.447440 17555 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.447448 17557 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.447580 17555 heartbeater.cc:507] Master 127.9.117.254:33847 requested a full tablet report, sending...
I20251023 20:12:40.447580 17557 heartbeater.cc:507] Master 127.9.117.253:36315 requested a full tablet report, sending...
I20251023 20:12:40.447953 17041 ts_manager.cc:194] Registered new tserver with Master: 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477)
I20251023 20:12:40.447942 16428 ts_manager.cc:194] Re-registered known tserver with Master: 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477)
I20251023 20:12:40.448454 16428 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:42901
I20251023 20:12:40.448575 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:33119
--local_ip_for_outbound_sockets=127.9.117.195
--tserver_master_addrs=127.9.117.254:33847
--webserver_port=34635
--webserver_interface=127.9.117.195
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315,127.9.117.252:34927 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:40.530373 17562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:40.530565 17562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:40.530606 17562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:40.532089 17562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:40.532131 17562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:40.532188 17562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:40.533850 17562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:33119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=34635
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33847,127.9.117.253:36315,127.9.117.252:34927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:40.534121 17562 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:40.534314 17562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:40.537149 17570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:40.537151 17568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:40.537227 17562 server_base.cc:1047] running on GCE node
W20251023 20:12:40.537364 17567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:40.537554 17562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:40.537796 17562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:40.538966 17562 hybrid_clock.cc:648] HybridClock initialized: now 1761250360538949 us; error 45 us; skew 500 ppm
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:40.541850 17562 init.cc:377] Logged in from keytab as oryx/127.9.117.195@KRBTEST.COM (short username oryx)
I20251023 20:12:40.542449 17562 webserver.cc:492] Webserver started at http://127.9.117.195:34635/ using document root <none> and password file <none>
I20251023 20:12:40.542654 17562 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:40.542699 17562 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:40.544078 17562 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:40.544808 17577 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.545028 17562 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:40.545109 17562 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "528e68b3955f41478e86fcd969d8e333"
format_stamp: "Formatted at 2025-10-23 20:12:38 on dist-test-slave-x62p"
server_key: "88977434670608788da67b3e8fb18978"
server_key_iv: "482ceb56c3f33411d676f87c44877460"
server_key_version: "encryptionkey@0"
I20251023 20:12:40.545406 17562 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:40.561463 17562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:40.563604 17562 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:40.563726 17562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:40.563944 17562 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:40.564266 17562 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:40.564301 17562 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.564323 17562 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:40.564337 17562 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:40.570299 17562 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:33119
I20251023 20:12:40.570395 17690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:33119 every 8 connection(s)
I20251023 20:12:40.570703 17562 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:12:40.573238 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 17562
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.252@KRBTEST.COM
Oct 23 20:12:40 dist-test-slave-x62p krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250360, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.253@KRBTEST.COM
I20251023 20:12:40.583495 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.571424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:36315 (local address 127.9.117.195:57271)
1023 20:12:40.571617 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.571650 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.572274 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.572521 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.572526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.573097 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.573614 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.573617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.574591 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.574592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.575256 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.575258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.576090 (+ 832us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.576647 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.576651 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.579307 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.581669 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.581679 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.581684 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.582065 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.582253 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.582254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.582255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.582347 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.582659 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.582666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.582919 (+ 253us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.583054 (+ 135us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.583228 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20251023 20:12:40.583992 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.571431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:57271 (local address 127.9.117.253:36315)
1023 20:12:40.571498 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.571501 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.571831 (+ 330us) server_negotiation.cc:408] Connection header received
1023 20:12:40.572448 (+ 617us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.572450 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.572459 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.572507 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.573721 (+ 1214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.574476 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.575360 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.575538 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.579403 (+ 3865us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.579409 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.579410 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.579418 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.581568 (+ 2150us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.582154 (+ 586us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.582156 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.582156 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.582185 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.582428 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.582429 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.582429 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.582558 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.582626 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.583777 (+ 1151us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.583888 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251023 20:12:40.584710 17693 heartbeater.cc:344] Connected to a master server at 127.9.117.253:36315
I20251023 20:12:40.584838 17693 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.585125 17693 heartbeater.cc:507] Master 127.9.117.253:36315 requested a full tablet report, sending...
I20251023 20:12:40.585757 17041 ts_manager.cc:194] Registered new tserver with Master: 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119)
I20251023 20:12:40.586352 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.574534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.9.117.253:36315)
1023 20:12:40.574900 (+ 366us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.574902 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.575823 (+ 921us) server_negotiation.cc:408] Connection header received
1023 20:12:40.576063 (+ 240us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.576064 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.576079 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.576155 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.578083 (+ 1928us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.578823 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.579818 (+ 995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.579997 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.582802 (+ 2805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.582808 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.582808 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.582814 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.584860 (+ 2046us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.585271 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.585272 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.585272 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.585297 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.585467 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.585468 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.585468 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.585612 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.585715 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.586139 (+ 424us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.586234 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":107,"threads_started":1}
I20251023 20:12:40.586680 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.571424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:34927 (local address 127.9.117.195:45577)
1023 20:12:40.571618 (+ 194us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.571649 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.572265 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.572520 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.572524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.573081 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.573554 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.573559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.574576 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.574577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.575235 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.575239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.575370 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.576134 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.576163 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.578789 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.585670 (+ 6881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.585671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.585671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.585847 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.586035 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.586036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.586036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.586068 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.586352 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.586353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.586395 (+ 42us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.586460 (+ 65us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.586588 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20251023 20:12:40.587021 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.571658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:45577 (local address 127.9.117.252:34927)
1023 20:12:40.571709 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.571711 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.571722 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:12:40.572376 (+ 654us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.572377 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.572392 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.572453 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.573687 (+ 1234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.574473 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.575621 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.575840 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.578906 (+ 3066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.578911 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.578912 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.578920 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.585583 (+ 6663us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.585939 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.585940 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.585940 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.585968 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.586129 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.586130 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.586130 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.586266 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.586324 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.586819 (+ 495us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.586918 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251023 20:12:40.587213 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.575585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.9.117.252:34927)
1023 20:12:40.575938 (+ 353us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.575940 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.577198 (+ 1258us) server_negotiation.cc:408] Connection header received
1023 20:12:40.577331 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.577333 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.577346 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.577436 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.580350 (+ 2914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.581223 (+ 873us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.582146 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.582369 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.583276 (+ 907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.583280 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.583280 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.583287 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.586054 (+ 2767us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.586392 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.586393 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.586393 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.586418 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.586583 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.586583 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.586584 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.586687 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.586781 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.586978 (+ 197us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.587151 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":97,"threads_started":1}
I20251023 20:12:40.587296 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.571874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:46533 (local address 127.9.117.254:33847)
1023 20:12:40.571951 (+ 77us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.571954 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.571997 (+ 43us) server_negotiation.cc:408] Connection header received
1023 20:12:40.572533 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.572535 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.572546 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.572615 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.573833 (+ 1218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.574558 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.575356 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.575537 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.579699 (+ 4162us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.579704 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.579704 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.579713 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.583435 (+ 3722us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.585508 (+ 2073us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.585509 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.585510 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.585544 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.586292 (+ 748us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.586294 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.586295 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.586414 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.586471 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.587081 (+ 610us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.587187 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251023 20:12:40.587399 17692 heartbeater.cc:344] Connected to a master server at 127.9.117.252:34927
I20251023 20:12:40.587455 17692 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.587564 17692 heartbeater.cc:507] Master 127.9.117.252:34927 requested a full tablet report, sending...
I20251023 20:12:40.587881 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.571722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:46533)
1023 20:12:40.571938 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.571947 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.572282 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.572631 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.572632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.573101 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:40.573724 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.573726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.574648 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.574649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.575256 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.575258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.575595 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.576378 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:40.576383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:40.579148 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:40.585129 (+ 5981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.585130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.585130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.585411 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.586188 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:40.586189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:40.586189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:40.586214 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:40.586912 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:40.586912 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:40.586952 (+ 40us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.587687 (+ 735us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.587808 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":62,"threads_started":1}
I20251023 20:12:40.587890 17222 ts_manager.cc:194] Registered new tserver with Master: 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119)
I20251023 20:12:40.588274 17691 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33847
I20251023 20:12:40.588331 17691 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:40.588423 17691 heartbeater.cc:507] Master 127.9.117.254:33847 requested a full tablet report, sending...
I20251023 20:12:40.588673 16428 ts_manager.cc:194] Re-registered known tserver with Master: 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119)
I20251023 20:12:40.589076 16428 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:46533
I20251023 20:12:40.590302 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.575139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.9.117.254:33847)
1023 20:12:40.576269 (+ 1130us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.576271 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.576284 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:12:40.576314 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.576314 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.576327 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.576382 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.577903 (+ 1521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.578670 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.579650 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.579855 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.580962 (+ 1107us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.580967 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.580967 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.580974 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.589337 (+ 8363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.589644 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.589645 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.589646 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.589670 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.589807 (+ 137us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.589807 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.589808 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.589895 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.590004 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.590109 (+ 105us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.590200 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":83,"threads_started":1}
I20251023 20:12:40.591950 16428 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:12:40.593304 16428 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:40.601226 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.598060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:33119 (local address 127.0.0.1:45028)
1023 20:12:40.598502 (+ 442us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.598515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.598592 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.598932 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.598933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.598935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.599120 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.599122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.600014 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.600015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.600968 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.600971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.601054 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.601061 (+ 7us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.601093 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.601138 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":72,"threads_started":1}
I20251023 20:12:40.601649 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.597871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:39439 (local address 127.0.0.1:32904)
1023 20:12:40.598004 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.598014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.598075 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.598764 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.598765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.598768 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.599022 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.599024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.600434 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.600436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.601453 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.601456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.601506 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.601509 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.601532 (+ 23us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.601586 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":86,"threads_started":1}
I20251023 20:12:40.601752 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.598181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.9.117.195:33119)
1023 20:12:40.598689 (+ 508us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.598702 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.598710 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:12:40.598749 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.598752 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.598769 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.598842 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.599196 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.599922 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.601063 (+ 1141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.601570 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.601627 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.601659 (+ 32us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.601689 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":94,"threads_started":1}
I20251023 20:12:40.602331 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.598220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40477 (local address 127.0.0.1:43314)
1023 20:12:40.598525 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.598532 (+ 7us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.598581 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.598908 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.598909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.598912 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.599118 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.599120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.600436 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.600436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.602095 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.602099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.602172 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.602178 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.602210 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.602253 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":65,"threads_started":1}
I20251023 20:12:40.602312 17625 tablet_service.cc:1505] Processing CreateTablet for tablet 65d3d88c793c418c8075e141fd456a0c (DEFAULT_TABLE table=test-table [id=291f404765b247c6a042c2a7fe516b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:40.602674 17625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d3d88c793c418c8075e141fd456a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:40.602998 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.598347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.9.117.194:40477)
1023 20:12:40.598723 (+ 376us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.598732 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.598743 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:12:40.598784 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.598785 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.598802 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.598899 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.599215 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.600342 (+ 1127us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.602193 (+ 1851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.602755 (+ 562us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.602816 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.602849 (+ 33us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.602910 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":76,"threads_started":1}
I20251023 20:12:40.603281 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.598003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.9.117.193:39439)
1023 20:12:40.598571 (+ 568us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.598582 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.598595 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:12:40.598645 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.598648 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.598671 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.598755 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.599109 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.600312 (+ 1203us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.602521 (+ 2209us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.603074 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.603130 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.603162 (+ 32us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.603208 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":99,"threads_started":1}
I20251023 20:12:40.603932 17353 tablet_service.cc:1505] Processing CreateTablet for tablet 65d3d88c793c418c8075e141fd456a0c (DEFAULT_TABLE table=test-table [id=291f404765b247c6a042c2a7fe516b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:40.604295 17353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d3d88c793c418c8075e141fd456a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:40.605479 17489 tablet_service.cc:1505] Processing CreateTablet for tablet 65d3d88c793c418c8075e141fd456a0c (DEFAULT_TABLE table=test-table [id=291f404765b247c6a042c2a7fe516b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:40.605768 17489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d3d88c793c418c8075e141fd456a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:40.606199 17720 tablet_bootstrap.cc:492] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: Bootstrap starting.
I20251023 20:12:40.607302 17720 tablet_bootstrap.cc:654] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:40.607465 17721 tablet_bootstrap.cc:492] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: Bootstrap starting.
I20251023 20:12:40.607808 17720 log.cc:826] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:40.608340 17721 tablet_bootstrap.cc:654] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:40.608635 17721 log.cc:826] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:40.608721 17720 tablet_bootstrap.cc:492] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: No bootstrap required, opened a new log
I20251023 20:12:40.608829 17720 ts_tablet_manager.cc:1403] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:12:40.609218 17724 tablet_bootstrap.cc:492] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Bootstrap starting.
I20251023 20:12:40.609444 17721 tablet_bootstrap.cc:492] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: No bootstrap required, opened a new log
I20251023 20:12:40.609515 17721 ts_tablet_manager.cc:1403] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:12:40.610323 17724 tablet_bootstrap.cc:654] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:40.610426 17720 raft_consensus.cc:359] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.610560 17720 raft_consensus.cc:385] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:40.610584 17720 raft_consensus.cc:740] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528e68b3955f41478e86fcd969d8e333, State: Initialized, Role: FOLLOWER
I20251023 20:12:40.610677 17720 consensus_queue.cc:260] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.610924 17720 ts_tablet_manager.cc:1434] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:12:40.610967 17691 heartbeater.cc:499] Master 127.9.117.254:33847 was elected leader, sending a full tablet report...
I20251023 20:12:40.610960 17724 log.cc:826] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:40.611508 17721 raft_consensus.cc:359] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.611682 17721 raft_consensus.cc:385] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:40.611716 17721 raft_consensus.cc:740] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5a4db60789429ba2b82ab3f1bff442, State: Initialized, Role: FOLLOWER
I20251023 20:12:40.611805 17721 consensus_queue.cc:260] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.611864 17724 tablet_bootstrap.cc:492] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: No bootstrap required, opened a new log
I20251023 20:12:40.611929 17724 ts_tablet_manager.cc:1403] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:12:40.612066 17721 ts_tablet_manager.cc:1434] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:12:40.612113 17419 heartbeater.cc:499] Master 127.9.117.254:33847 was elected leader, sending a full tablet report...
I20251023 20:12:40.613806 17724 raft_consensus.cc:359] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.613952 17724 raft_consensus.cc:385] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:40.614008 17724 raft_consensus.cc:740] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71cd94153d6a4376935a19eec2ea6b3b, State: Initialized, Role: FOLLOWER
I20251023 20:12:40.614125 17724 consensus_queue.cc:260] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.614377 17724 ts_tablet_manager.cc:1434] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:12:40.614497 17555 heartbeater.cc:499] Master 127.9.117.254:33847 was elected leader, sending a full tablet report...
W20251023 20:12:40.683104 17558 tablet.cc:2378] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:12:40.693739 17727 raft_consensus.cc:493] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:40.693874 17727 raft_consensus.cc:515] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.694249 17727 leader_election.cc:290] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119), 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477)
I20251023 20:12:40.696529 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.694537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40477 (local address 127.9.117.193:40085)
1023 20:12:40.694655 (+ 118us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.694666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.694699 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.694860 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.694861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.694863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.695050 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.695052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.695702 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.695704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.696291 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.696293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.696366 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.696371 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.696404 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.696443 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20251023 20:12:40.696534 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.694431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:33119 (local address 127.9.117.193:43683)
1023 20:12:40.694489 (+ 58us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:40.694500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:40.694548 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:40.694842 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:40.694844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:40.694848 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:40.695084 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.695086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.695735 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.695736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:40.696304 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:40.696306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.696376 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.696380 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:12:40.696413 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:12:40.696452 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20251023 20:12:40.697027 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.694591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:43683 (local address 127.9.117.195:33119)
1023 20:12:40.694715 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.694718 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.694729 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:12:40.694760 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.694761 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.694771 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.694833 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.695171 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.695660 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.696391 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.696921 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.696930 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.696951 (+ 21us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.696978 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20251023 20:12:40.697046 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.694666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:40085 (local address 127.9.117.194:40477)
1023 20:12:40.694727 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.694730 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.694739 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:40.694768 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.694769 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.694780 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.694830 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:40.695128 (+ 298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.695625 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.696381 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.696922 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.696932 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.696956 (+ 24us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.696992 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251023 20:12:40.697396 17645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65d3d88c793c418c8075e141fd456a0c" candidate_uuid: "7b5a4db60789429ba2b82ab3f1bff442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528e68b3955f41478e86fcd969d8e333" is_pre_election: true
I20251023 20:12:40.697397 17509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65d3d88c793c418c8075e141fd456a0c" candidate_uuid: "7b5a4db60789429ba2b82ab3f1bff442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71cd94153d6a4376935a19eec2ea6b3b" is_pre_election: true
I20251023 20:12:40.697549 17645 raft_consensus.cc:2468] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5a4db60789429ba2b82ab3f1bff442 in term 0.
I20251023 20:12:40.697541 17509 raft_consensus.cc:2468] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5a4db60789429ba2b82ab3f1bff442 in term 0.
I20251023 20:12:40.697830 17309 leader_election.cc:304] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71cd94153d6a4376935a19eec2ea6b3b, 7b5a4db60789429ba2b82ab3f1bff442; no voters:
I20251023 20:12:40.697978 17727 raft_consensus.cc:2804] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:40.698053 17727 raft_consensus.cc:493] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:40.698094 17727 raft_consensus.cc:3060] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:40.698899 17727 raft_consensus.cc:515] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.699040 17727 leader_election.cc:290] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [CANDIDATE]: Term 1 election: Requested vote from peers 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119), 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477)
I20251023 20:12:40.699345 17509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65d3d88c793c418c8075e141fd456a0c" candidate_uuid: "7b5a4db60789429ba2b82ab3f1bff442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71cd94153d6a4376935a19eec2ea6b3b"
I20251023 20:12:40.699354 17645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65d3d88c793c418c8075e141fd456a0c" candidate_uuid: "7b5a4db60789429ba2b82ab3f1bff442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528e68b3955f41478e86fcd969d8e333"
I20251023 20:12:40.699440 17645 raft_consensus.cc:3060] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:40.699440 17509 raft_consensus.cc:3060] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:40.700395 17645 raft_consensus.cc:2468] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5a4db60789429ba2b82ab3f1bff442 in term 1.
I20251023 20:12:40.700395 17509 raft_consensus.cc:2468] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5a4db60789429ba2b82ab3f1bff442 in term 1.
I20251023 20:12:40.700604 17309 leader_election.cc:304] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528e68b3955f41478e86fcd969d8e333, 7b5a4db60789429ba2b82ab3f1bff442; no voters:
I20251023 20:12:40.700711 17727 raft_consensus.cc:2804] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:40.700906 17727 raft_consensus.cc:697] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 1 LEADER]: Becoming Leader. State: Replica: 7b5a4db60789429ba2b82ab3f1bff442, State: Running, Role: LEADER
I20251023 20:12:40.701112 17727 consensus_queue.cc:237] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } }
I20251023 20:12:40.702001 16428 catalog_manager.cc:5649] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b5a4db60789429ba2b82ab3f1bff442 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "7b5a4db60789429ba2b82ab3f1bff442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } health_report { overall_health: UNKNOWN } } }
I20251023 20:12:40.722594 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.720377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.9.117.193:39439)
1023 20:12:40.720454 (+ 77us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.720457 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.720466 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:40.720517 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.720518 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.720538 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.720594 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:40.720925 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.721429 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.722026 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.722187 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.722234 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:40.722343 (+ 109us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:40.722422 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.722472 (+ 50us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.722499 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:12:40.724522 17645 raft_consensus.cc:1275] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 1 FOLLOWER]: Refusing update from remote peer 7b5a4db60789429ba2b82ab3f1bff442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:40.724584 17509 raft_consensus.cc:1275] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 1 FOLLOWER]: Refusing update from remote peer 7b5a4db60789429ba2b82ab3f1bff442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:40.724941 17729 consensus_queue.cc:1048] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:40.725076 17727 consensus_queue.cc:1048] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:40.728315 17733 mvcc.cc:204] Tried to move back new op lower bound from 7214081477524623360 to 7214081477432451072. Current Snapshot: MvccSnapshot[applied={T|T < 7214081477524623360}]
I20251023 20:12:40.728652 17734 mvcc.cc:204] Tried to move back new op lower bound from 7214081477524623360 to 7214081477432451072. Current Snapshot: MvccSnapshot[applied={T|T < 7214081477524623360}]
I20251023 20:12:40.731699 16428 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42334:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:40.731805 16428 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42334:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:40.733711 16428 catalog_manager.cc:5936] T 00000000000000000000000000000000 P f53480ec5ac94aed91148cbc54a3e99f: Sending DeleteTablet for 3 replicas of tablet 65d3d88c793c418c8075e141fd456a0c
I20251023 20:12:40.734097 17353 tablet_service.cc:1552] Processing DeleteTablet for tablet 65d3d88c793c418c8075e141fd456a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:40 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:32904
I20251023 20:12:40.734089 17625 tablet_service.cc:1552] Processing DeleteTablet for tablet 65d3d88c793c418c8075e141fd456a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:40 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:45028
I20251023 20:12:40.734413 17489 tablet_service.cc:1552] Processing DeleteTablet for tablet 65d3d88c793c418c8075e141fd456a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:40 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:43314
I20251023 20:12:40.734800 17743 tablet_replica.cc:333] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: stopping tablet replica
I20251023 20:12:40.734864 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 16397
I20251023 20:12:40.735800 17744 tablet_replica.cc:333] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: stopping tablet replica
I20251023 20:12:40.735908 17744 raft_consensus.cc:2243] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:40.736027 17744 raft_consensus.cc:2272] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:40.736294 17742 tablet_replica.cc:333] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: stopping tablet replica
I20251023 20:12:40.736377 17742 raft_consensus.cc:2243] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:12:40.736508 17742 raft_consensus.cc:2272] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:40.736869 17744 ts_tablet_manager.cc:1916] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:40.739022 17744 ts_tablet_manager.cc:1929] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:40.739100 17744 log.cc:1199] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/65d3d88c793c418c8075e141fd456a0c
I20251023 20:12:40.739379 17744 ts_tablet_manager.cc:1950] T 65d3d88c793c418c8075e141fd456a0c P 71cd94153d6a4376935a19eec2ea6b3b: Deleting consensus metadata
I20251023 20:12:40.740137 17742 ts_tablet_manager.cc:1916] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:40.742486 17742 ts_tablet_manager.cc:1929] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:40.742568 17742 log.cc:1199] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/65d3d88c793c418c8075e141fd456a0c
I20251023 20:12:40.742874 17742 ts_tablet_manager.cc:1950] T 65d3d88c793c418c8075e141fd456a0c P 7b5a4db60789429ba2b82ab3f1bff442: Deleting consensus metadata
W20251023 20:12:40.742914 17443 connection.cc:537] server connection from 127.0.0.1:43314 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:12:40.743274 17309 connection.cc:441] server connection from 127.0.0.1:32904 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32904 (request call id 1) could send its response
I20251023 20:12:40.734925 17743 raft_consensus.cc:2243] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:40.744122 17743 raft_consensus.cc:2272] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:40.745218 17743 ts_tablet_manager.cc:1916] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:40.746780 17743 ts_tablet_manager.cc:1929] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:40.746865 17743 log.cc:1199] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/65d3d88c793c418c8075e141fd456a0c
I20251023 20:12:40.747120 17743 ts_tablet_manager.cc:1950] T 65d3d88c793c418c8075e141fd456a0c P 528e68b3955f41478e86fcd969d8e333: Deleting consensus metadata
W20251023 20:12:40.747728 17578 connection.cc:441] server connection from 127.0.0.1:45028 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45028 (request call id 1) could send its response
I20251023 20:12:40.756233 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.745143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.9.117.253:36315)
1023 20:12:40.745203 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.745205 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.748946 (+ 3741us) server_negotiation.cc:408] Connection header received
1023 20:12:40.749031 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.749032 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.749052 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.749127 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.750134 (+ 1007us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.750690 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.751331 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.751500 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.753290 (+ 1790us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.753295 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.753295 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.753304 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.755075 (+ 1771us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.755377 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.755378 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.755378 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.755401 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.755537 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.755538 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.755539 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.755671 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.755782 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.756021 (+ 239us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.756108 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251023 20:12:40.756706 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:40.745464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.9.117.252:34927)
1023 20:12:40.745546 (+ 82us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:40.745549 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:40.749264 (+ 3715us) server_negotiation.cc:408] Connection header received
1023 20:12:40.749315 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:40.749317 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:40.749338 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:40.749432 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:40.750204 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.751011 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:40.752349 (+ 1338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:40.752558 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:40.753312 (+ 754us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:40.753318 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:40.753318 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:40.753326 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:40.755415 (+ 2089us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.755907 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.755908 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.755908 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.755936 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:40.756130 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:40.756131 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:40.756131 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:40.756242 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:40.756316 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:40.756517 (+ 201us) server_negotiation.cc:300] Negotiation successful
1023 20:12:40.756612 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251023 20:12:41.730005 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:41.729418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:58961)
1023 20:12:41.729660 (+ 242us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:41.729816 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":72,"threads_started":1}
I20251023 20:12:41.730181 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:41.729752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:49935)
1023 20:12:41.729974 (+ 222us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:41.730053 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":71,"threads_started":1}
W20251023 20:12:41.730314 17555 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:33847 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:41.730321 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:41.729976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:33407)
1023 20:12:41.730197 (+ 221us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:41.730272 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":55,"threads_started":1}
W20251023 20:12:41.730356 17419 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:33847 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
W20251023 20:12:41.730448 17691 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:33847 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:42.233686 17755 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f53480ec5ac94aed91148cbc54a3e99f)
I20251023 20:12:42.233891 17756 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f53480ec5ac94aed91148cbc54a3e99f)
I20251023 20:12:42.233870 17755 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } }
I20251023 20:12:42.234030 17756 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } }
I20251023 20:12:42.234163 17755 leader_election.cc:290] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847), 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252:34927)
I20251023 20:12:42.234308 17756 leader_election.cc:290] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847), 4ab6da4f96394fada9592954ac0c7e69 (127.9.117.253:36315)
I20251023 20:12:42.234555 17233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ab6da4f96394fada9592954ac0c7e69" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1eaa0286b9bc4f4594974bceb95a3741" is_pre_election: true
I20251023 20:12:42.234539 17051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1eaa0286b9bc4f4594974bceb95a3741" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4ab6da4f96394fada9592954ac0c7e69" is_pre_election: true
I20251023 20:12:42.234649 17233 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab6da4f96394fada9592954ac0c7e69 in term 1.
I20251023 20:12:42.234649 17051 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaa0286b9bc4f4594974bceb95a3741 in term 1.
I20251023 20:12:42.234671 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.234414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42816)
1023 20:12:42.234581 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.234623 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":56,"threads_started":1}
I20251023 20:12:42.234535 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.234315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42804)
1023 20:12:42.234427 (+ 112us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.234482 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":48,"threads_started":1}
W20251023 20:12:42.234782 17208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:42.234807 17207 leader_election.cc:304] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaa0286b9bc4f4594974bceb95a3741, 4ab6da4f96394fada9592954ac0c7e69; no voters:
I20251023 20:12:42.234840 17026 leader_election.cc:304] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaa0286b9bc4f4594974bceb95a3741, 4ab6da4f96394fada9592954ac0c7e69; no voters:
I20251023 20:12:42.234922 17756 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 20:12:42.234946 17755 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 20:12:42.235004 17755 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f53480ec5ac94aed91148cbc54a3e99f)
I20251023 20:12:42.235024 17755 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:12:42.234968 17756 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f53480ec5ac94aed91148cbc54a3e99f)
I20251023 20:12:42.235095 17756 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:12:42.235303 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.235161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42826)
1023 20:12:42.235253 (+ 92us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.235279 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20251023 20:12:42.235411 17208 leader_election.cc:336] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847): Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:42.236147 17756 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } }
I20251023 20:12:42.236151 17755 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } }
I20251023 20:12:42.236294 17756 leader_election.cc:290] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [CANDIDATE]: Term 2 election: Requested vote from peers f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847), 4ab6da4f96394fada9592954ac0c7e69 (127.9.117.253:36315)
W20251023 20:12:42.234958 17026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:42.236502 17051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1eaa0286b9bc4f4594974bceb95a3741" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4ab6da4f96394fada9592954ac0c7e69"
I20251023 20:12:42.236599 17051 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eaa0286b9bc4f4594974bceb95a3741 in current term 2: Already voted for candidate 4ab6da4f96394fada9592954ac0c7e69 in this term.
I20251023 20:12:42.236572 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.236392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42830)
1023 20:12:42.236468 (+ 76us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.236547 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20251023 20:12:42.236275 17755 leader_election.cc:290] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 2 election: Requested vote from peers f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847), 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252:34927)
I20251023 20:12:42.236835 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.236638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42842)
1023 20:12:42.236785 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.236811 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20251023 20:12:42.236900 17026 leader_election.cc:336] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847): Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:42.237228 17233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ab6da4f96394fada9592954ac0c7e69" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1eaa0286b9bc4f4594974bceb95a3741"
I20251023 20:12:42.237316 17233 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ab6da4f96394fada9592954ac0c7e69 in current term 2: Already voted for candidate 1eaa0286b9bc4f4594974bceb95a3741 in this term.
I20251023 20:12:42.237332 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.237249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42860)
1023 20:12:42.237287 (+ 38us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.237306 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20251023 20:12:42.237413 17026 leader_election.cc:336] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847): Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:42.237481 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.237215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:42848)
1023 20:12:42.237251 (+ 36us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.237272 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20251023 20:12:42.237484 17026 leader_election.cc:304] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ab6da4f96394fada9592954ac0c7e69; no voters: 1eaa0286b9bc4f4594974bceb95a3741, f53480ec5ac94aed91148cbc54a3e99f
I20251023 20:12:42.237599 17755 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251023 20:12:42.237640 17208 leader_election.cc:336] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847): Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:42.237686 17208 leader_election.cc:304] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eaa0286b9bc4f4594974bceb95a3741; no voters: 4ab6da4f96394fada9592954ac0c7e69, f53480ec5ac94aed91148cbc54a3e99f
I20251023 20:12:42.237782 17756 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251023 20:12:42.730995 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.730691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:34383)
1023 20:12:42.730884 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.730931 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20251023 20:12:42.731244 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.730754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:49395)
1023 20:12:42.731155 (+ 401us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.731191 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":91,"threads_started":1}
I20251023 20:12:42.731408 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:42.731103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:59499)
1023 20:12:42.731315 (+ 212us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:42.731349 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":55,"threads_started":1}
I20251023 20:12:43.731992 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:43.731486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:41207)
1023 20:12:43.731877 (+ 391us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:43.731923 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":86,"threads_started":1}
I20251023 20:12:43.732453 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:43.731819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:34173)
1023 20:12:43.732376 (+ 557us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:43.732413 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":105,"threads_started":1}
I20251023 20:12:43.732482 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:43.731975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:55263)
1023 20:12:43.732396 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:43.732434 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":50,"threads_started":1}
I20251023 20:12:44.033121 17767 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:44.033247 17767 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } }
I20251023 20:12:44.033473 17767 leader_election.cc:290] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847), 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252:34927)
I20251023 20:12:44.033842 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.033639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:43082)
1023 20:12:44.033755 (+ 116us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.033799 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20251023 20:12:44.033941 17233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ab6da4f96394fada9592954ac0c7e69" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1eaa0286b9bc4f4594974bceb95a3741" is_pre_election: true
I20251023 20:12:44.034044 17233 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab6da4f96394fada9592954ac0c7e69 in term 2.
I20251023 20:12:44.034200 17026 leader_election.cc:304] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaa0286b9bc4f4594974bceb95a3741, 4ab6da4f96394fada9592954ac0c7e69; no voters:
I20251023 20:12:44.034253 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.034149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:43088)
1023 20:12:44.034182 (+ 33us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.034207 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20251023 20:12:44.034293 17026 leader_election.cc:336] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847): Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:44.034328 17767 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 20:12:44.034380 17767 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:44.034415 17767 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 2 FOLLOWER]: Advancing to term 3
I20251023 20:12:44.035228 17767 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } }
I20251023 20:12:44.035346 17767 leader_election.cc:290] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 3 election: Requested vote from peers f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847), 1eaa0286b9bc4f4594974bceb95a3741 (127.9.117.252:34927)
I20251023 20:12:44.035535 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.035444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:43094)
1023 20:12:44.035486 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.035513 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20251023 20:12:44.035648 17233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ab6da4f96394fada9592954ac0c7e69" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1eaa0286b9bc4f4594974bceb95a3741"
I20251023 20:12:44.035737 17233 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 2 FOLLOWER]: Advancing to term 3
I20251023 20:12:44.035866 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.035777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:43102)
1023 20:12:44.035814 (+ 37us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.035840 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20251023 20:12:44.035940 17026 leader_election.cc:336] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847): Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
I20251023 20:12:44.036526 17233 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab6da4f96394fada9592954ac0c7e69 in term 3.
I20251023 20:12:44.036706 17026 leader_election.cc:304] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eaa0286b9bc4f4594974bceb95a3741, 4ab6da4f96394fada9592954ac0c7e69; no voters: f53480ec5ac94aed91148cbc54a3e99f
I20251023 20:12:44.036818 17767 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 20:12:44.036921 17767 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [term 3 LEADER]: Becoming Leader. State: Replica: 4ab6da4f96394fada9592954ac0c7e69, State: Running, Role: LEADER
I20251023 20:12:44.036993 17767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } }
I20251023 20:12:44.037389 17770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ab6da4f96394fada9592954ac0c7e69. Latest consensus state: current_term: 3 leader_uuid: "4ab6da4f96394fada9592954ac0c7e69" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:44.037485 17770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:44.037693 17772 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:44.038039 17772 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:44.038100 17772 catalog_manager.cc:1269] Loaded cluster ID: b9ef92cf951f4d268d65e7afaf5f4080
I20251023 20:12:44.038121 17772 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:44.038272 17772 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:44.038328 17772 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Loaded TSK: 0
I20251023 20:12:44.038439 17772 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:12:44.065433 17039 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:12:44.067595 17039 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:44.069972 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.069547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:43124)
1023 20:12:44.069731 (+ 184us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.069897 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20251023 20:12:44.070429 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.070236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.0.0.1:43130)
1023 20:12:44.070365 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.070394 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
W20251023 20:12:44.070513 17026 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 -> Peer f53480ec5ac94aed91148cbc54a3e99f (127.9.117.254:33847): Couldn't send request to peer f53480ec5ac94aed91148cbc54a3e99f. Status: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251023 20:12:44.070636 17233 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [term 3 FOLLOWER]: Refusing update from remote peer 4ab6da4f96394fada9592954ac0c7e69: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251023 20:12:44.070886 17767 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251023 20:12:44.071688 17775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ab6da4f96394fada9592954ac0c7e69. Latest consensus state: current_term: 3 leader_uuid: "4ab6da4f96394fada9592954ac0c7e69" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:44.071825 17775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:12:44.072283 17770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4ab6da4f96394fada9592954ac0c7e69" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:44.072391 17770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:44.072494 17773 mvcc.cc:204] Tried to move back new op lower bound from 7214081491223797760 to 7214081491096285184. Current Snapshot: MvccSnapshot[applied={T|T < 7214081491223797760}]
I20251023 20:12:44.072467 17775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4ab6da4f96394fada9592954ac0c7e69" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f53480ec5ac94aed91148cbc54a3e99f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33847 } } peers { permanent_uuid: "4ab6da4f96394fada9592954ac0c7e69" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 36315 } attrs { promote: false } } peers { permanent_uuid: "1eaa0286b9bc4f4594974bceb95a3741" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 34927 } attrs { promote: false } } }
I20251023 20:12:44.072576 17775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaa0286b9bc4f4594974bceb95a3741 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:12:44.078589 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.075132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:33119 (local address 127.0.0.1:45040)
1023 20:12:44.075351 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.075363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:44.075415 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:44.075602 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:44.075603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:44.075605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:44.075908 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:44.075911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.077133 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:44.077134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:44.078267 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:44.078270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.078331 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:44.078336 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:12:44.078363 (+ 27us) client_negotiation.cc:240] Negotiation successful
1023 20:12:44.078424 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20251023 20:12:44.078522 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.075218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:39439 (local address 127.0.0.1:32922)
1023 20:12:44.075438 (+ 220us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.075448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:44.075497 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:44.076221 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:44.076222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:44.076224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:44.076403 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:44.076405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.077352 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:44.077353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:44.078266 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:44.078269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.078319 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:44.078325 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:12:44.078350 (+ 25us) client_negotiation.cc:240] Negotiation successful
1023 20:12:44.078391 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20251023 20:12:44.079034 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.075031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40477 (local address 127.0.0.1:43330)
1023 20:12:44.075192 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.075206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:44.075272 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:44.076002 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:44.076002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:44.076004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:44.076223 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:44.076225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.077179 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:44.077180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:44.078369 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:44.078370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.078428 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:44.078430 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:12:44.078462 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:12:44.078498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20251023 20:12:44.079504 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.075275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.9.117.195:33119)
1023 20:12:44.075397 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:44.075400 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:44.075419 (+ 19us) server_negotiation.cc:408] Connection header received
1023 20:12:44.075491 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:44.075492 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:44.075506 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:44.075982 (+ 476us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:44.076042 (+ 60us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.077016 (+ 974us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:44.078646 (+ 1630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.079291 (+ 645us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:44.079303 (+ 12us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:44.079331 (+ 28us) server_negotiation.cc:300] Negotiation successful
1023 20:12:44.079385 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20251023 20:12:44.079797 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.075585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.9.117.193:39439)
1023 20:12:44.076082 (+ 497us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:44.076085 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:44.076102 (+ 17us) server_negotiation.cc:408] Connection header received
1023 20:12:44.076127 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:44.076128 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:44.076139 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:44.076468 (+ 329us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:44.076520 (+ 52us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.077271 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:44.078902 (+ 1631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.079517 (+ 615us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:44.079526 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:44.079555 (+ 29us) server_negotiation.cc:300] Negotiation successful
1023 20:12:44.079724 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":58,"threads_started":1}
I20251023 20:12:44.079811 17625 tablet_service.cc:1505] Processing CreateTablet for tablet dc9540331f8a42a2a7d2082e7a94080a (DEFAULT_TABLE table=test-table [id=00e551255da447cc93503ab3c50d4462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:44.079964 17625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9540331f8a42a2a7d2082e7a94080a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:44.080435 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.075114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.9.117.194:40477)
1023 20:12:44.075846 (+ 732us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:44.075849 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:44.075867 (+ 18us) server_negotiation.cc:408] Connection header received
1023 20:12:44.075894 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:44.075895 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:44.075905 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:44.075984 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:44.076309 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.077085 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:44.078590 (+ 1505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.080316 (+ 1726us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:44.080324 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:44.080347 (+ 23us) server_negotiation.cc:300] Negotiation successful
1023 20:12:44.080379 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":65,"threads_started":1}
I20251023 20:12:44.080943 17489 tablet_service.cc:1505] Processing CreateTablet for tablet dc9540331f8a42a2a7d2082e7a94080a (DEFAULT_TABLE table=test-table [id=00e551255da447cc93503ab3c50d4462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:44.081092 17489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9540331f8a42a2a7d2082e7a94080a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:44.082342 17786 tablet_bootstrap.cc:492] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: Bootstrap starting.
I20251023 20:12:44.082362 17353 tablet_service.cc:1505] Processing CreateTablet for tablet dc9540331f8a42a2a7d2082e7a94080a (DEFAULT_TABLE table=test-table [id=00e551255da447cc93503ab3c50d4462]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:44.082492 17353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9540331f8a42a2a7d2082e7a94080a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:44.082865 17786 tablet_bootstrap.cc:654] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:44.083637 17786 tablet_bootstrap.cc:492] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: No bootstrap required, opened a new log
I20251023 20:12:44.083705 17786 ts_tablet_manager.cc:1403] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:44.083906 17786 raft_consensus.cc:359] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.084002 17786 raft_consensus.cc:385] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:44.084028 17786 raft_consensus.cc:740] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71cd94153d6a4376935a19eec2ea6b3b, State: Initialized, Role: FOLLOWER
I20251023 20:12:44.084136 17786 consensus_queue.cc:260] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.084340 17786 ts_tablet_manager.cc:1434] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:44.084365 17785 tablet_bootstrap.cc:492] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: Bootstrap starting.
I20251023 20:12:44.084721 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.084592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:40981)
1023 20:12:44.084648 (+ 56us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.084681 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20251023 20:12:44.084888 17785 tablet_bootstrap.cc:654] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:44.085533 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.085412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:56431)
1023 20:12:44.085477 (+ 65us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.085501 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20251023 20:12:44.085640 17785 tablet_bootstrap.cc:492] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: No bootstrap required, opened a new log
I20251023 20:12:44.085693 17785 ts_tablet_manager.cc:1403] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:44.085744 17557 heartbeater.cc:499] Master 127.9.117.253:36315 was elected leader, sending a full tablet report...
I20251023 20:12:44.085878 17785 raft_consensus.cc:359] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.085963 17785 raft_consensus.cc:385] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:44.085983 17785 raft_consensus.cc:740] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528e68b3955f41478e86fcd969d8e333, State: Initialized, Role: FOLLOWER
I20251023 20:12:44.086042 17785 consensus_queue.cc:260] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.086059 17788 tablet_bootstrap.cc:492] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442: Bootstrap starting.
I20251023 20:12:44.086229 17785 ts_tablet_manager.cc:1434] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:44.086653 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.086496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:34471)
1023 20:12:44.086586 (+ 90us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.086615 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20251023 20:12:44.086820 17693 heartbeater.cc:499] Master 127.9.117.253:36315 was elected leader, sending a full tablet report...
I20251023 20:12:44.087105 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.087006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:56607)
1023 20:12:44.087046 (+ 40us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.087078 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20251023 20:12:44.087121 17788 tablet_bootstrap.cc:654] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:44.089293 17788 tablet_bootstrap.cc:492] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442: No bootstrap required, opened a new log
I20251023 20:12:44.089376 17788 ts_tablet_manager.cc:1403] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:12:44.089550 17788 raft_consensus.cc:359] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.089640 17788 raft_consensus.cc:385] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:44.089668 17788 raft_consensus.cc:740] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5a4db60789429ba2b82ab3f1bff442, State: Initialized, Role: FOLLOWER
I20251023 20:12:44.089771 17788 consensus_queue.cc:260] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.089999 17788 ts_tablet_manager.cc:1434] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:44.090381 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.090277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:49625)
1023 20:12:44.090317 (+ 40us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.090347 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20251023 20:12:44.093031 17421 heartbeater.cc:499] Master 127.9.117.253:36315 was elected leader, sending a full tablet report...
I20251023 20:12:44.093205 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.093080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:42753)
1023 20:12:44.093130 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.093165 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20251023 20:12:44.306082 17422 tablet.cc:2378] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:12:44.324486 17694 tablet.cc:2378] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:12:44.357267 17793 raft_consensus.cc:493] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:44.357409 17793 raft_consensus.cc:515] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.357626 17793 leader_election.cc:290] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477), 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119)
I20251023 20:12:44.357920 17509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc9540331f8a42a2a7d2082e7a94080a" candidate_uuid: "7b5a4db60789429ba2b82ab3f1bff442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71cd94153d6a4376935a19eec2ea6b3b" is_pre_election: true
I20251023 20:12:44.357950 17645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc9540331f8a42a2a7d2082e7a94080a" candidate_uuid: "7b5a4db60789429ba2b82ab3f1bff442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528e68b3955f41478e86fcd969d8e333" is_pre_election: true
I20251023 20:12:44.358036 17645 raft_consensus.cc:2468] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5a4db60789429ba2b82ab3f1bff442 in term 0.
I20251023 20:12:44.358040 17509 raft_consensus.cc:2468] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5a4db60789429ba2b82ab3f1bff442 in term 0.
I20251023 20:12:44.358208 17309 leader_election.cc:304] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71cd94153d6a4376935a19eec2ea6b3b, 7b5a4db60789429ba2b82ab3f1bff442; no voters:
I20251023 20:12:44.358361 17793 raft_consensus.cc:2804] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:44.358433 17793 raft_consensus.cc:493] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:44.358454 17793 raft_consensus.cc:3060] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:44.359400 17793 raft_consensus.cc:515] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.359526 17793 leader_election.cc:290] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [CANDIDATE]: Term 1 election: Requested vote from peers 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477), 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119)
I20251023 20:12:44.359730 17509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc9540331f8a42a2a7d2082e7a94080a" candidate_uuid: "7b5a4db60789429ba2b82ab3f1bff442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71cd94153d6a4376935a19eec2ea6b3b"
I20251023 20:12:44.359771 17645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc9540331f8a42a2a7d2082e7a94080a" candidate_uuid: "7b5a4db60789429ba2b82ab3f1bff442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528e68b3955f41478e86fcd969d8e333"
I20251023 20:12:44.359807 17509 raft_consensus.cc:3060] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:44.359848 17645 raft_consensus.cc:3060] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:44.360446 17509 raft_consensus.cc:2468] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5a4db60789429ba2b82ab3f1bff442 in term 1.
I20251023 20:12:44.360466 17645 raft_consensus.cc:2468] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5a4db60789429ba2b82ab3f1bff442 in term 1.
I20251023 20:12:44.360659 17309 leader_election.cc:304] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528e68b3955f41478e86fcd969d8e333, 7b5a4db60789429ba2b82ab3f1bff442; no voters:
I20251023 20:12:44.360853 17793 raft_consensus.cc:2804] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:44.361029 17793 raft_consensus.cc:697] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [term 1 LEADER]: Becoming Leader. State: Replica: 7b5a4db60789429ba2b82ab3f1bff442, State: Running, Role: LEADER
I20251023 20:12:44.361091 17793 consensus_queue.cc:237] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } }
I20251023 20:12:44.361385 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.361265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:42683)
1023 20:12:44.361315 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.361349 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20251023 20:12:44.362053 17039 catalog_manager.cc:5649] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b5a4db60789429ba2b82ab3f1bff442 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "7b5a4db60789429ba2b82ab3f1bff442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b5a4db60789429ba2b82ab3f1bff442" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 39439 } health_report { overall_health: HEALTHY } } }
I20251023 20:12:44.433369 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.431003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.9.117.193:39439)
1023 20:12:44.431074 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:44.431077 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:44.431085 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:12:44.431110 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:44.431111 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:44.431121 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:44.431169 (+ 48us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:44.431506 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.432276 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:44.432928 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:44.433056 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:44.433074 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:44.433100 (+ 26us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:44.433148 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:44.433255 (+ 107us) server_negotiation.cc:300] Negotiation successful
1023 20:12:44.433279 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251023 20:12:44.434410 17645 raft_consensus.cc:1275] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 1 FOLLOWER]: Refusing update from remote peer 7b5a4db60789429ba2b82ab3f1bff442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:44.434410 17509 raft_consensus.cc:1275] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 1 FOLLOWER]: Refusing update from remote peer 7b5a4db60789429ba2b82ab3f1bff442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:44.434779 17793 consensus_queue.cc:1048] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71cd94153d6a4376935a19eec2ea6b3b" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:44.434922 17794 consensus_queue.cc:1048] T dc9540331f8a42a2a7d2082e7a94080a P 7b5a4db60789429ba2b82ab3f1bff442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528e68b3955f41478e86fcd969d8e333" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:44.435556 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.435435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:55247)
1023 20:12:44.435482 (+ 47us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.435513 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20251023 20:12:44.436061 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.435925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:43519)
1023 20:12:44.436001 (+ 76us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.436028 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20251023 20:12:44.437016 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.436680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:56085)
1023 20:12:44.436950 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.436981 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20251023 20:12:44.437160 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.436976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.194:51089)
1023 20:12:44.437105 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.437131 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20251023 20:12:44.438122 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.437980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.195:50213)
1023 20:12:44.438050 (+ 70us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.438090 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20251023 20:12:44.438843 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:44.437345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33847 (local address 127.9.117.193:56937)
1023 20:12:44.438779 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.438814 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1415}
I20251023 20:12:44.439314 17039 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40900:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:44.439435 17039 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40900:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:44.440820 17039 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 4ab6da4f96394fada9592954ac0c7e69: Sending DeleteTablet for 3 replicas of tablet dc9540331f8a42a2a7d2082e7a94080a
I20251023 20:12:44.441098 17489 tablet_service.cc:1552] Processing DeleteTablet for tablet dc9540331f8a42a2a7d2082e7a94080a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:44 UTC) from {username='oryx', principal='oryx/127.9.117.253@KRBTEST.COM'} at 127.0.0.1:43330
I20251023 20:12:44.441151 17625 tablet_service.cc:1552] Processing DeleteTablet for tablet dc9540331f8a42a2a7d2082e7a94080a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:44 UTC) from {username='oryx', principal='oryx/127.9.117.253@KRBTEST.COM'} at 127.0.0.1:45040
I20251023 20:12:44.441260 17353 tablet_service.cc:1552] Processing DeleteTablet for tablet dc9540331f8a42a2a7d2082e7a94080a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:44 UTC) from {username='oryx', principal='oryx/127.9.117.253@KRBTEST.COM'} at 127.0.0.1:32922
I20251023 20:12:44.441357 17805 tablet_replica.cc:333] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: stopping tablet replica
I20251023 20:12:44.441421 17805 raft_consensus.cc:2243] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:44.441471 17805 raft_consensus.cc:2272] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:44.441612 17806 tablet_replica.cc:333] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: stopping tablet replica
I20251023 20:12:44.441644 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 17290
I20251023 20:12:44.441679 17806 raft_consensus.cc:2243] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:44.441730 17806 raft_consensus.cc:2272] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:44.442656 17806 ts_tablet_manager.cc:1916] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:44.444226 17806 ts_tablet_manager.cc:1929] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:44.444279 17806 log.cc:1199] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/dc9540331f8a42a2a7d2082e7a94080a
I20251023 20:12:44.444525 17806 ts_tablet_manager.cc:1950] T dc9540331f8a42a2a7d2082e7a94080a P 528e68b3955f41478e86fcd969d8e333: Deleting consensus metadata
I20251023 20:12:44.444885 17028 catalog_manager.cc:4985] TS 528e68b3955f41478e86fcd969d8e333 (127.9.117.195:33119): tablet dc9540331f8a42a2a7d2082e7a94080a (table test-table [id=00e551255da447cc93503ab3c50d4462]) successfully deleted
I20251023 20:12:44.445346 17805 ts_tablet_manager.cc:1916] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:44.446532 17805 ts_tablet_manager.cc:1929] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:44.446607 17805 log.cc:1199] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/dc9540331f8a42a2a7d2082e7a94080a
I20251023 20:12:44.446810 17805 ts_tablet_manager.cc:1950] T dc9540331f8a42a2a7d2082e7a94080a P 71cd94153d6a4376935a19eec2ea6b3b: Deleting consensus metadata
I20251023 20:12:44.447162 17028 catalog_manager.cc:4985] TS 71cd94153d6a4376935a19eec2ea6b3b (127.9.117.194:40477): tablet dc9540331f8a42a2a7d2082e7a94080a (table test-table [id=00e551255da447cc93503ab3c50d4462]) successfully deleted
W20251023 20:12:44.448814 17774 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:44.447981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:39439 (local address 127.0.0.1:32948)
1023 20:12:44.448027 (+ 46us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:44.448037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:44.448080 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:44.448722 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:39439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
I20251023 20:12:44.448904 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 17426
W20251023 20:12:44.448927 17028 catalog_manager.cc:4712] TS 7b5a4db60789429ba2b82ab3f1bff442 (127.9.117.193:39439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc9540331f8a42a2a7d2082e7a94080a: Network error: Client connection negotiation failed: client connection to 127.9.117.193:39439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:12:44.454142 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 17562
I20251023 20:12:44.458940 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 16901
I20251023 20:12:44.464258 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 17022
2025-10-23T20:12:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8804 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:44 dist-test-slave-x62p krb5kdc[17810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:44 dist-test-slave-x62p krb5kdc[17810](info): set up 2 sockets
Oct 23 20:12:44 dist-test-slave-x62p krb5kdc[17810](info): commencing operation
krb5kdc: starting...
W20251023 20:12:46.461988 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:46 dist-test-slave-x62p krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:46.608419 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43291
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:43291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:46.692835 17826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:46.693028 17826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:46.693068 17826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:46.694523 17826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:46.694582 17826 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251023 20:12:46.694608 17826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:46.694628 17826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:46.694649 17826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:46.694681 17826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:46.696334 17826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:43291
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:46.696624 17826 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:46.696847 17826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:12:46.699872 17826 server_base.cc:1047] running on GCE node
W20251023 20:12:46.699873 17834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:46.699879 17832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:46.699883 17831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:46.700392 17826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:46.700634 17826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:46.701771 17826 hybrid_clock.cc:648] HybridClock initialized: now 1761250366701746 us; error 43 us; skew 500 ppm
Oct 23 20:12:46 dist-test-slave-x62p krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250366, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:46.704092 17826 init.cc:377] Logged in from keytab as oryx/127.9.117.254@KRBTEST.COM (short username oryx)
I20251023 20:12:46.704694 17826 webserver.cc:492] Webserver started at http://127.9.117.254:38279/ using document root <none> and password file <none>
I20251023 20:12:46.704932 17826 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:46.704974 17826 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:46.705076 17826 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:46.706130 17826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "16bc966345ee4ab9a689d4ea11589e35"
format_stamp: "Formatted at 2025-10-23 20:12:46 on dist-test-slave-x62p"
server_key: "de03e6c16be297fa1f1c6e01ec764942"
server_key_iv: "e24b42d86c4eede037d4c4d05ef3edbd"
server_key_version: "encryptionkey@0"
I20251023 20:12:46.706456 17826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "16bc966345ee4ab9a689d4ea11589e35"
format_stamp: "Formatted at 2025-10-23 20:12:46 on dist-test-slave-x62p"
server_key: "de03e6c16be297fa1f1c6e01ec764942"
server_key_iv: "e24b42d86c4eede037d4c4d05ef3edbd"
server_key_version: "encryptionkey@0"
I20251023 20:12:46.707864 17826 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:46.708701 17841 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:46.708953 17826 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:46.709035 17826 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "16bc966345ee4ab9a689d4ea11589e35"
format_stamp: "Formatted at 2025-10-23 20:12:46 on dist-test-slave-x62p"
server_key: "de03e6c16be297fa1f1c6e01ec764942"
server_key_iv: "e24b42d86c4eede037d4c4d05ef3edbd"
server_key_version: "encryptionkey@0"
I20251023 20:12:46.709113 17826 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:46.728345 17826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:46.730332 17826 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:46.730480 17826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:46.735412 17826 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:43291
I20251023 20:12:46.735496 17903 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:43291 every 8 connection(s)
I20251023 20:12:46.735800 17826 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:46.736680 17904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:46.738955 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: Bootstrap starting.
I20251023 20:12:46.739516 17904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:46.739742 17904 log.cc:826] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:46.740461 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: No bootstrap required, opened a new log
I20251023 20:12:46.741756 17904 raft_consensus.cc:359] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc966345ee4ab9a689d4ea11589e35" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43291 } }
I20251023 20:12:46.741904 17904 raft_consensus.cc:385] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:46.741925 17904 raft_consensus.cc:740] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bc966345ee4ab9a689d4ea11589e35, State: Initialized, Role: FOLLOWER
I20251023 20:12:46.742033 17904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc966345ee4ab9a689d4ea11589e35" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43291 } }
I20251023 20:12:46.742103 17904 raft_consensus.cc:399] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:46.742137 17904 raft_consensus.cc:493] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:46.742170 17904 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:46.742708 17904 raft_consensus.cc:515] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc966345ee4ab9a689d4ea11589e35" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43291 } }
I20251023 20:12:46.742811 17904 leader_election.cc:304] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16bc966345ee4ab9a689d4ea11589e35; no voters:
I20251023 20:12:46.742944 17904 leader_election.cc:290] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:46.743005 17907 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:46.743175 17907 raft_consensus.cc:697] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [term 1 LEADER]: Becoming Leader. State: Replica: 16bc966345ee4ab9a689d4ea11589e35, State: Running, Role: LEADER
I20251023 20:12:46.743182 17904 sys_catalog.cc:565] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:46.743291 17907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc966345ee4ab9a689d4ea11589e35" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43291 } }
I20251023 20:12:46.743434 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 17826
I20251023 20:12:46.743538 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:46.743721 9687 external_mini_cluster.cc:1468] Setting key f429cceb41c8bdd03536442bc65c6368
I20251023 20:12:46.744627 17909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16bc966345ee4ab9a689d4ea11589e35. Latest consensus state: current_term: 1 leader_uuid: "16bc966345ee4ab9a689d4ea11589e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc966345ee4ab9a689d4ea11589e35" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43291 } } }
I20251023 20:12:46.744716 17909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:46.744915 17908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16bc966345ee4ab9a689d4ea11589e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bc966345ee4ab9a689d4ea11589e35" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43291 } } }
I20251023 20:12:46.745005 17908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:46.745141 17919 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:46.745628 17919 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:46.747185 17919 catalog_manager.cc:1357] Generated new cluster ID: e1105f6e3e944b20a6fc3be1d0910c75
I20251023 20:12:46.747238 17919 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:12:46 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:46.763520 17919 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:46.764161 17919 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:46.765012 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:46.744539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.9.117.254:43291)
1023 20:12:46.744712 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:46.744746 (+ 34us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:46.744772 (+ 26us) server_negotiation.cc:408] Connection header received
1023 20:12:46.744896 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:46.744903 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:46.744959 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:46.746022 (+ 1063us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:46.746116 (+ 94us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.747078 (+ 962us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:46.757730 (+ 10652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.758045 (+ 315us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:46.760886 (+ 2841us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:46.760898 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:46.760898 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:46.760932 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:46.763201 (+ 2269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:46.763670 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:46.763674 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:46.763675 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:46.763755 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:46.763964 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:46.763965 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:46.763965 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:46.764218 (+ 253us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:46.764364 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:46.764641 (+ 277us) server_negotiation.cc:300] Negotiation successful
1023 20:12:46.764823 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20251023 20:12:46.770803 17919 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: Generated new TSK 0
I20251023 20:12:46.771024 17919 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:46.827934 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43291
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:46.912822 17932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:46.913012 17932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:46.913041 17932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:46.914600 17932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:46.914655 17932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:46.914678 17932 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:12:46.914696 17932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:46.916330 17932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:43291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:46.916589 17932 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:46.916826 17932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:46.919709 17937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:46.919713 17938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:46.919821 17932 server_base.cc:1047] running on GCE node
W20251023 20:12:46.919713 17940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:46.920107 17932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:46.920302 17932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:46.921439 17932 hybrid_clock.cc:648] HybridClock initialized: now 1761250366921423 us; error 29 us; skew 500 ppm
Oct 23 20:12:46 dist-test-slave-x62p krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250366, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:46.923784 17932 init.cc:377] Logged in from keytab as oryx/127.9.117.193@KRBTEST.COM (short username oryx)
I20251023 20:12:46.924333 17932 webserver.cc:492] Webserver started at http://127.9.117.193:44603/ using document root <none> and password file <none>
I20251023 20:12:46.924505 17932 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:46.924537 17932 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:46.924649 17932 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:46.925668 17932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "1f49bf0f5f714183aa61273df0068c3f"
format_stamp: "Formatted at 2025-10-23 20:12:46 on dist-test-slave-x62p"
server_key: "9d1408d6f0c75f479317d0a691e1d0c2"
server_key_iv: "60c3c052575bc206c589b00c1b672434"
server_key_version: "encryptionkey@0"
I20251023 20:12:46.925940 17932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "1f49bf0f5f714183aa61273df0068c3f"
format_stamp: "Formatted at 2025-10-23 20:12:46 on dist-test-slave-x62p"
server_key: "9d1408d6f0c75f479317d0a691e1d0c2"
server_key_iv: "60c3c052575bc206c589b00c1b672434"
server_key_version: "encryptionkey@0"
I20251023 20:12:46.927085 17932 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:46.927719 17947 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:46.927902 17932 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:46.927968 17932 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "1f49bf0f5f714183aa61273df0068c3f"
format_stamp: "Formatted at 2025-10-23 20:12:46 on dist-test-slave-x62p"
server_key: "9d1408d6f0c75f479317d0a691e1d0c2"
server_key_iv: "60c3c052575bc206c589b00c1b672434"
server_key_version: "encryptionkey@0"
I20251023 20:12:46.928040 17932 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:46.945305 17932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:46.947158 17932 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:46.947284 17932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:46.947880 17932 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:46.947929 17932 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:46.947962 17932 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:46.947990 17932 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 23 20:12:46 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250366, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:46.955794 17932 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:38553
I20251023 20:12:46.955873 18062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:38553 every 8 connection(s)
I20251023 20:12:46.956205 17932 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:46.957031 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:46.948225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:37299 (local address 127.9.117.254:43291)
1023 20:12:46.948282 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:46.948284 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:46.948468 (+ 184us) server_negotiation.cc:408] Connection header received
1023 20:12:46.949089 (+ 621us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:46.949091 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:46.949105 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:46.949162 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:46.949984 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.950508 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:46.951175 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.951369 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:46.953854 (+ 2485us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:46.953860 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:46.953861 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:46.953872 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:46.955505 (+ 1633us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:46.956006 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:46.956008 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:46.956008 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:46.956039 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:46.956271 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:46.956271 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:46.956272 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:46.956388 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:46.956459 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:46.956853 (+ 394us) server_negotiation.cc:300] Negotiation successful
1023 20:12:46.956962 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:12:46.957120 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:46.948204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43291 (local address 127.9.117.193:37299)
1023 20:12:46.948365 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:46.948392 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:46.948970 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:46.949245 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:46.949249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:46.949559 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:46.949880 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:46.949883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.950606 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:46.950607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:46.951083 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:46.951085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.951223 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:46.951900 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:46.951932 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:46.953736 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:46.955606 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:46.955615 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:46.955618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:46.955915 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:46.956105 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:46.956106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:46.956107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:46.956195 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:46.956460 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:46.956465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:46.956643 (+ 178us) client_negotiation.cc:769] Sending connection context
1023 20:12:46.956833 (+ 190us) client_negotiation.cc:240] Negotiation successful
1023 20:12:46.956973 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20251023 20:12:46.957810 18063 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43291
I20251023 20:12:46.957911 18063 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:46.958122 18063 heartbeater.cc:507] Master 127.9.117.254:43291 requested a full tablet report, sending...
I20251023 20:12:46.958600 17858 ts_manager.cc:194] Registered new tserver with Master: 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553)
I20251023 20:12:46.959553 17858 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:37299
I20251023 20:12:46.963282 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 17932
I20251023 20:12:46.963395 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:46.963666 9687 external_mini_cluster.cc:1468] Setting key b73e22fcdaed756db93dfa8cbbcbfae8
I20251023 20:12:46.964238 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:46.958926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:50937 (local address 127.9.117.254:43291)
1023 20:12:46.958971 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:46.958973 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:46.959052 (+ 79us) server_negotiation.cc:408] Connection header received
1023 20:12:46.959122 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:46.959123 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:46.959130 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:46.959176 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:46.959759 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.960255 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:46.960798 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.960938 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:46.961580 (+ 642us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:46.961583 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:46.961584 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:46.961590 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:46.963231 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:46.963585 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:46.963587 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:46.963587 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:46.963617 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:46.963764 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:46.963765 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:46.963765 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:46.963882 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:46.963951 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:46.964094 (+ 143us) server_negotiation.cc:300] Negotiation successful
1023 20:12:46.964175 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:12:46.964269 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:46.958848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43291 (local address 127.9.117.193:50937)
1023 20:12:46.958989 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:46.959000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:46.959067 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:46.959190 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:46.959191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:46.959352 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:46.959593 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:46.959596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.960330 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:46.960331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:46.960680 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:46.960682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:46.960778 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:46.961243 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:46.961247 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:46.961495 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:46.963338 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:46.963338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:46.963339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:46.963514 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:46.963686 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:46.963687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:46.963687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:46.963708 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:46.963948 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:46.963949 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:46.963997 (+ 48us) client_negotiation.cc:769] Sending connection context
1023 20:12:46.964055 (+ 58us) client_negotiation.cc:240] Negotiation successful
1023 20:12:46.964163 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:47.011040 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43291
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:47.094198 18074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:47.094396 18074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:47.094439 18074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:47.095965 18074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:47.096019 18074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:47.096055 18074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:12:47.096086 18074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:47.097780 18074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:43291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:47.098062 18074 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:47.098263 18074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:47.101048 18082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:47.101074 18079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:47.101117 18080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:47.101322 18074 server_base.cc:1047] running on GCE node
I20251023 20:12:47.101480 18074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:47.101676 18074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:47.102826 18074 hybrid_clock.cc:648] HybridClock initialized: now 1761250367102814 us; error 28 us; skew 500 ppm
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250367, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:47.105453 18074 init.cc:377] Logged in from keytab as oryx/127.9.117.194@KRBTEST.COM (short username oryx)
I20251023 20:12:47.106083 18074 webserver.cc:492] Webserver started at http://127.9.117.194:44595/ using document root <none> and password file <none>
I20251023 20:12:47.106295 18074 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:47.106333 18074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:47.106456 18074 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:47.107498 18074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "e934ee0612b44f4aaebb9063a6bb284e"
format_stamp: "Formatted at 2025-10-23 20:12:47 on dist-test-slave-x62p"
server_key: "be14c8707812521ad852e63bcd688ab3"
server_key_iv: "c27f506e65496fd91725ef5b1af76b0d"
server_key_version: "encryptionkey@0"
I20251023 20:12:47.107844 18074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "e934ee0612b44f4aaebb9063a6bb284e"
format_stamp: "Formatted at 2025-10-23 20:12:47 on dist-test-slave-x62p"
server_key: "be14c8707812521ad852e63bcd688ab3"
server_key_iv: "c27f506e65496fd91725ef5b1af76b0d"
server_key_version: "encryptionkey@0"
I20251023 20:12:47.109212 18074 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:47.110021 18089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.110214 18074 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:47.110286 18074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "e934ee0612b44f4aaebb9063a6bb284e"
format_stamp: "Formatted at 2025-10-23 20:12:47 on dist-test-slave-x62p"
server_key: "be14c8707812521ad852e63bcd688ab3"
server_key_iv: "c27f506e65496fd91725ef5b1af76b0d"
server_key_version: "encryptionkey@0"
I20251023 20:12:47.110355 18074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:47.121726 18074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:47.123875 18074 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:47.123999 18074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:47.124577 18074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:47.124624 18074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.124666 18074 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:47.124694 18074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250367, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:47.132622 18074 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:40033
I20251023 20:12:47.132714 18204 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:40033 every 8 connection(s)
I20251023 20:12:47.133044 18074 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:12:47.134616 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.125017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:45845 (local address 127.9.117.254:43291)
1023 20:12:47.125089 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.125091 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.125268 (+ 177us) server_negotiation.cc:408] Connection header received
1023 20:12:47.126078 (+ 810us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.126079 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.126090 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.126135 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.127357 (+ 1222us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.127912 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.128873 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.129071 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.131626 (+ 2555us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:47.131630 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:47.131631 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:47.131639 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:47.133292 (+ 1653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.133769 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.133770 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.133771 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.133802 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.134035 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.134036 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.134036 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.134145 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:47.134216 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.134445 (+ 229us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.134543 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251023 20:12:47.134678 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.124966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43291 (local address 127.9.117.194:45845)
1023 20:12:47.125129 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.125169 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.125974 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.126208 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.126213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.126716 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:47.127226 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.127233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.127994 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.127996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.128705 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.128708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.128946 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.129629 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:47.129662 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:47.131513 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:47.133447 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.133452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.133455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.133660 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.133891 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.133892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.133892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.133950 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.134209 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:47.134213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:47.134335 (+ 122us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.134420 (+ 85us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.134530 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20251023 20:12:47.135205 18205 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43291
I20251023 20:12:47.135291 18205 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:47.135500 18205 heartbeater.cc:507] Master 127.9.117.254:43291 requested a full tablet report, sending...
I20251023 20:12:47.135787 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 18074
I20251023 20:12:47.135880 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:47.135939 17857 ts_manager.cc:194] Registered new tserver with Master: e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033)
I20251023 20:12:47.136072 9687 external_mini_cluster.cc:1468] Setting key 943ee25a52387830f278cc11e742a099
I20251023 20:12:47.136504 17857 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:45845
I20251023 20:12:47.142953 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.136452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:43669 (local address 127.9.117.254:43291)
1023 20:12:47.136490 (+ 38us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.136492 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.136764 (+ 272us) server_negotiation.cc:408] Connection header received
1023 20:12:47.137347 (+ 583us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.137348 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.137361 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.137456 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.137869 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.138579 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.139374 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.139488 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.140386 (+ 898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:47.140390 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:47.140391 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:47.140397 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:47.141830 (+ 1433us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.142209 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.142210 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.142211 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.142230 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.142381 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.142382 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.142382 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.142496 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:47.142555 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.142792 (+ 237us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.142883 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20251023 20:12:47.142968 18211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.136420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43291 (local address 127.9.117.194:43669)
1023 20:12:47.136668 (+ 248us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.136680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.136766 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.137469 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.137470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.137609 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:47.137786 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.137788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.138660 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.138661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.139290 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.139293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.139374 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.139862 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:47.139867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:47.140282 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:47.141930 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.141932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.141932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.142133 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.142288 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.142289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.142289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.142325 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.142555 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:47.142556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:47.142620 (+ 64us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.142701 (+ 81us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.142820 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab.
I20251023 20:12:47.185792 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43291
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:47.268965 18216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:47.269155 18216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:47.269182 18216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:47.270679 18216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:47.270746 18216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:47.270782 18216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251023 20:12:47.270815 18216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:47.272500 18216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/krb5kdc/oryx.keytab
--principal=oryx/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.117.254:43291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:47.272720 18216 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:47.272933 18216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:47.275465 18224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:47.275477 18222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:47.275516 18221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:47.275763 18216 server_base.cc:1047] running on GCE node
I20251023 20:12:47.275942 18216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:47.276152 18216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:47.277308 18216 hybrid_clock.cc:648] HybridClock initialized: now 1761250367277289 us; error 31 us; skew 500 ppm
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250367, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:47.279976 18216 init.cc:377] Logged in from keytab as oryx/127.9.117.195@KRBTEST.COM (short username oryx)
I20251023 20:12:47.280578 18216 webserver.cc:492] Webserver started at http://127.9.117.195:44921/ using document root <none> and password file <none>
I20251023 20:12:47.280824 18216 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:47.280866 18216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:47.280967 18216 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:47.281983 18216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "9353b0ea583f4e069ce9ff52104c09ba"
format_stamp: "Formatted at 2025-10-23 20:12:47 on dist-test-slave-x62p"
server_key: "05754bddecb996cac4584f1093045a60"
server_key_iv: "c2ad0c0aa8db799c04700acb6d9cb57d"
server_key_version: "encryptionkey@0"
I20251023 20:12:47.282270 18216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "9353b0ea583f4e069ce9ff52104c09ba"
format_stamp: "Formatted at 2025-10-23 20:12:47 on dist-test-slave-x62p"
server_key: "05754bddecb996cac4584f1093045a60"
server_key_iv: "c2ad0c0aa8db799c04700acb6d9cb57d"
server_key_version: "encryptionkey@0"
I20251023 20:12:47.283499 18216 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:47.284291 18231 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.284475 18216 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:47.284551 18216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "9353b0ea583f4e069ce9ff52104c09ba"
format_stamp: "Formatted at 2025-10-23 20:12:47 on dist-test-slave-x62p"
server_key: "05754bddecb996cac4584f1093045a60"
server_key_iv: "c2ad0c0aa8db799c04700acb6d9cb57d"
server_key_version: "encryptionkey@0"
I20251023 20:12:47.284616 18216 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:47.295416 18216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:47.297271 18216 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:47.297379 18216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:47.297909 18216 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:47.297945 18216 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.297968 18216 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:47.297981 18216 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250367, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:47.305135 18216 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:34851
I20251023 20:12:47.305209 18346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:34851 every 8 connection(s)
I20251023 20:12:47.305541 18216 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:12:47.307377 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.298319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:41183 (local address 127.9.117.254:43291)
1023 20:12:47.298387 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.298389 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.298654 (+ 265us) server_negotiation.cc:408] Connection header received
1023 20:12:47.299198 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.299199 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.299211 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.299278 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.300509 (+ 1231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.301399 (+ 890us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.302085 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.302266 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.304434 (+ 2168us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:47.304440 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:47.304440 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:47.304449 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:47.306172 (+ 1723us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.306636 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.306637 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.306637 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.306660 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.306844 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.306845 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.306846 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.306951 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:47.307017 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.307237 (+ 220us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.307310 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251023 20:12:47.307499 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.298279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43291 (local address 127.9.117.195:41183)
1023 20:12:47.298552 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.298576 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.299096 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.299338 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.299343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.299933 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:47.300390 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.300395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.301506 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.301507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.301997 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.301999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.302148 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.302694 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:47.302712 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:47.304316 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:47.306317 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.306321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.306324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.306524 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.306708 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.306709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.306709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.306763 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.307017 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:47.307020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:47.307135 (+ 115us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.307217 (+ 82us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.307328 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":92,"threads_started":1}
I20251023 20:12:47.307998 18347 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43291
I20251023 20:12:47.308089 18347 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:47.308281 18347 heartbeater.cc:507] Master 127.9.117.254:43291 requested a full tablet report, sending...
I20251023 20:12:47.308686 17857 ts_manager.cc:194] Registered new tserver with Master: 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851)
I20251023 20:12:47.309134 17857 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:41183
I20251023 20:12:47.310577 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 18216
I20251023 20:12:47.310658 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:47.310809 9687 external_mini_cluster.cc:1468] Setting key 2f5f61f7c693bce0ee72653ab92e704a
I20251023 20:12:47.311630 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:12:47.314483 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.309029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43291 (local address 127.9.117.195:34435)
1023 20:12:47.309171 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.309182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.309255 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.309372 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.309373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.309541 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:47.309720 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.309722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.310469 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.310470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.310953 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.310955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.311021 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.311631 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:47.311635 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:47.311926 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:47.313631 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.313632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.313633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.313777 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.313926 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.313926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.313926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.313949 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.314163 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:47.314164 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:47.314242 (+ 78us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.314310 (+ 68us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.314404 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20251023 20:12:47.314594 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.309106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:34435 (local address 127.9.117.254:43291)
1023 20:12:47.309171 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.309173 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.309237 (+ 64us) server_negotiation.cc:408] Connection header received
1023 20:12:47.309309 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.309310 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.309320 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.309370 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.309921 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.310391 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.311020 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.311216 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.312001 (+ 785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:47.312004 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:47.312004 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:47.312011 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:47.313506 (+ 1495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.313844 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.313845 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.313846 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.313871 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.313987 (+ 116us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.313988 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.313989 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.314105 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:47.314172 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.314368 (+ 196us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.314455 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250367, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
W20251023 20:12:47.332691 17918 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:47.327280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.9.117.254:43291)
1023 20:12:47.327351 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.327353 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.327365 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:12:47.327398 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.327399 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.327407 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.327459 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.328207 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.328707 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.329399 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.329597 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.332596 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50142: BlockingRecv error: recv got EOF from 127.0.0.1:50142 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250367, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:47.339776 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.333535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.9.117.254:43291)
1023 20:12:47.333580 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.333581 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.333655 (+ 74us) server_negotiation.cc:408] Connection header received
1023 20:12:47.333716 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.333716 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.333725 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.333787 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.334346 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.334818 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.335299 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.335400 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.337531 (+ 2131us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:47.337536 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:47.337536 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:47.337543 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:47.338934 (+ 1391us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.339258 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.339258 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.339259 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.339275 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.339382 (+ 107us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.339382 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.339382 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.339471 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:47.339548 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.339640 (+ 92us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.339705 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251023 20:12:47.341351 17857 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:12:47.342590 17857 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:47.348165 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.344838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40033 (local address 127.0.0.1:39298)
1023 20:12:47.345652 (+ 814us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.345663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.345728 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.345907 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.345908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.345910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.346120 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.346122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.347034 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.347036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.347852 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.347854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.347929 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.347983 (+ 54us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.348036 (+ 53us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.348077 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":78,"threads_started":1}
I20251023 20:12:47.348925 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.344838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34851 (local address 127.0.0.1:46350)
1023 20:12:47.345806 (+ 968us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.345816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.345862 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.346186 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.346187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.346189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.346417 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.346418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.347680 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.347682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.348637 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.348640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.348706 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.348712 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.348805 (+ 93us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.348848 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":6,"thread_start_us":119,"threads_started":1}
I20251023 20:12:47.349212 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.345109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.9.117.194:40033)
1023 20:12:47.345293 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.345301 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.345737 (+ 436us) server_negotiation.cc:408] Connection header received
1023 20:12:47.345796 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.345799 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.345820 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.345895 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.346189 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.346934 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.348356 (+ 1422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.348968 (+ 612us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.349035 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.349071 (+ 36us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.349127 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20251023 20:12:47.349263 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.344835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:38553 (local address 127.0.0.1:54752)
1023 20:12:47.345121 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.345184 (+ 63us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.345275 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.345557 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.345562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.345566 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.346050 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.346054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.347414 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.347415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.348251 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.348254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.349073 (+ 819us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.349078 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.349146 (+ 68us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.349191 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20251023 20:12:47.349565 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.344939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.9.117.193:38553)
1023 20:12:47.345125 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.345134 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.345357 (+ 223us) server_negotiation.cc:408] Connection header received
1023 20:12:47.345410 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.345413 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.345432 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.345495 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.346123 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.347309 (+ 1186us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.348393 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.348969 (+ 576us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.349035 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.349196 (+ 161us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.349260 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20251023 20:12:47.349989 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.345167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.9.117.195:34851)
1023 20:12:47.345380 (+ 213us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.345388 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.345937 (+ 549us) server_negotiation.cc:408] Connection header received
1023 20:12:47.345984 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.345987 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.346005 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.346066 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.346499 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.347577 (+ 1078us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.348809 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.349779 (+ 970us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.349823 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.349858 (+ 35us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.349911 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
I20251023 20:12:47.350000 18139 tablet_service.cc:1505] Processing CreateTablet for tablet 88b158cf055745899aa61268552e9d20 (DEFAULT_TABLE table=test-table [id=d12fd007f2294d37aa14683931b7b9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.350200 17997 tablet_service.cc:1505] Processing CreateTablet for tablet 88b158cf055745899aa61268552e9d20 (DEFAULT_TABLE table=test-table [id=d12fd007f2294d37aa14683931b7b9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.350360 18139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b158cf055745899aa61268552e9d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.350502 17997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b158cf055745899aa61268552e9d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.350605 18281 tablet_service.cc:1505] Processing CreateTablet for tablet 88b158cf055745899aa61268552e9d20 (DEFAULT_TABLE table=test-table [id=d12fd007f2294d37aa14683931b7b9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.350857 18281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b158cf055745899aa61268552e9d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.352697 18375 tablet_bootstrap.cc:492] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Bootstrap starting.
I20251023 20:12:47.353385 18375 tablet_bootstrap.cc:654] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.353500 18376 tablet_bootstrap.cc:492] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: Bootstrap starting.
I20251023 20:12:47.353817 18377 tablet_bootstrap.cc:492] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: Bootstrap starting.
I20251023 20:12:47.354245 18376 tablet_bootstrap.cc:654] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.354395 18377 tablet_bootstrap.cc:654] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.354398 18375 log.cc:826] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:47.354885 18376 log.cc:826] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:47.355068 18377 log.cc:826] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:47.355402 18375 tablet_bootstrap.cc:492] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: No bootstrap required, opened a new log
I20251023 20:12:47.355485 18375 ts_tablet_manager.cc:1403] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:12:47.355650 18376 tablet_bootstrap.cc:492] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: No bootstrap required, opened a new log
I20251023 20:12:47.355717 18376 ts_tablet_manager.cc:1403] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:12:47.355847 18377 tablet_bootstrap.cc:492] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: No bootstrap required, opened a new log
I20251023 20:12:47.355927 18377 ts_tablet_manager.cc:1403] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:12:47.357365 18376 raft_consensus.cc:359] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.357316 18375 raft_consensus.cc:359] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.357517 18375 raft_consensus.cc:385] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.357520 18376 raft_consensus.cc:385] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.357550 18375 raft_consensus.cc:740] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e934ee0612b44f4aaebb9063a6bb284e, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.357553 18376 raft_consensus.cc:740] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f49bf0f5f714183aa61273df0068c3f, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.357679 18376 consensus_queue.cc:260] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.357646 18377 raft_consensus.cc:359] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.357772 18377 raft_consensus.cc:385] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.357728 18375 consensus_queue.cc:260] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.357803 18377 raft_consensus.cc:740] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9353b0ea583f4e069ce9ff52104c09ba, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.357986 18376 ts_tablet_manager.cc:1434] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:12:47.357988 18375 ts_tablet_manager.cc:1434] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251023 20:12:47.357950 18377 consensus_queue.cc:260] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.358132 18063 heartbeater.cc:499] Master 127.9.117.254:43291 was elected leader, sending a full tablet report...
I20251023 20:12:47.358211 18377 ts_tablet_manager.cc:1434] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:12:47.358242 18347 heartbeater.cc:499] Master 127.9.117.254:43291 was elected leader, sending a full tablet report...
I20251023 20:12:47.358446 18205 heartbeater.cc:499] Master 127.9.117.254:43291 was elected leader, sending a full tablet report...
W20251023 20:12:47.383598 18206 tablet.cc:2378] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:12:47.429950 18381 raft_consensus.cc:493] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:47.430085 18381 raft_consensus.cc:515] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.430392 18381 leader_election.cc:290] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033), 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851)
I20251023 20:12:47.432861 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.430592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40033 (local address 127.9.117.193:53801)
1023 20:12:47.430641 (+ 49us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.430654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.430704 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.430908 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.430910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.430912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.431105 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.431106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.431979 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.431980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.432632 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.432634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.432694 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.432698 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.432722 (+ 24us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.432788 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20251023 20:12:47.432983 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.430594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34851 (local address 127.9.117.193:55025)
1023 20:12:47.430891 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.430899 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.430963 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.431092 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.431092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.431094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.431416 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.431418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.432144 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.432145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.432722 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.432746 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.432830 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.432833 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.432867 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.432905 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":167,"thread_start_us":69,"threads_started":1}
I20251023 20:12:47.433379 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.430698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:53801 (local address 127.9.117.194:40033)
1023 20:12:47.430757 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.430760 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.430771 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:12:47.430804 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.430804 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.430816 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.430898 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.431194 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.431878 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.432747 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.433242 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.433252 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.433275 (+ 23us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.433310 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:12:47.433413 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.430689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:55025 (local address 127.9.117.195:34851)
1023 20:12:47.430760 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.430763 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.430970 (+ 207us) server_negotiation.cc:408] Connection header received
1023 20:12:47.431020 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.431022 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.431031 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.431085 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.431503 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.432060 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.432820 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.433283 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.433294 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.433318 (+ 24us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.433358 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:12:47.433702 18159 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b158cf055745899aa61268552e9d20" candidate_uuid: "1f49bf0f5f714183aa61273df0068c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e934ee0612b44f4aaebb9063a6bb284e" is_pre_election: true
I20251023 20:12:47.433701 18301 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b158cf055745899aa61268552e9d20" candidate_uuid: "1f49bf0f5f714183aa61273df0068c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9353b0ea583f4e069ce9ff52104c09ba" is_pre_election: true
I20251023 20:12:47.433838 18159 raft_consensus.cc:2468] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f49bf0f5f714183aa61273df0068c3f in term 0.
I20251023 20:12:47.433846 18301 raft_consensus.cc:2468] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f49bf0f5f714183aa61273df0068c3f in term 0.
I20251023 20:12:47.434054 17949 leader_election.cc:304] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f49bf0f5f714183aa61273df0068c3f, 9353b0ea583f4e069ce9ff52104c09ba; no voters:
I20251023 20:12:47.434190 18381 raft_consensus.cc:2804] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:47.434247 18381 raft_consensus.cc:493] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:47.434273 18381 raft_consensus.cc:3060] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.434998 18381 raft_consensus.cc:515] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.435128 18381 leader_election.cc:290] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [CANDIDATE]: Term 1 election: Requested vote from peers e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033), 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851)
I20251023 20:12:47.435297 18301 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b158cf055745899aa61268552e9d20" candidate_uuid: "1f49bf0f5f714183aa61273df0068c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9353b0ea583f4e069ce9ff52104c09ba"
I20251023 20:12:47.435310 18159 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b158cf055745899aa61268552e9d20" candidate_uuid: "1f49bf0f5f714183aa61273df0068c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e934ee0612b44f4aaebb9063a6bb284e"
I20251023 20:12:47.435377 18159 raft_consensus.cc:3060] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.435380 18301 raft_consensus.cc:3060] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.436049 18159 raft_consensus.cc:2468] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f49bf0f5f714183aa61273df0068c3f in term 1.
I20251023 20:12:47.436118 18301 raft_consensus.cc:2468] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f49bf0f5f714183aa61273df0068c3f in term 1.
I20251023 20:12:47.436244 17951 leader_election.cc:304] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f49bf0f5f714183aa61273df0068c3f, e934ee0612b44f4aaebb9063a6bb284e; no voters:
I20251023 20:12:47.436352 18381 raft_consensus.cc:2804] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:47.436535 18381 raft_consensus.cc:697] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 1 LEADER]: Becoming Leader. State: Replica: 1f49bf0f5f714183aa61273df0068c3f, State: Running, Role: LEADER
I20251023 20:12:47.436666 18381 consensus_queue.cc:237] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.437520 17857 catalog_manager.cc:5649] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193). New cstate: current_term: 1 leader_uuid: "1f49bf0f5f714183aa61273df0068c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } health_report { overall_health: UNKNOWN } } }
I20251023 20:12:47.446056 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.443839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.9.117.193:38553)
1023 20:12:47.443905 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.443907 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.443914 (+ 7us) server_negotiation.cc:408] Connection header received
1023 20:12:47.443989 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.443990 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.444006 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.444094 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:47.444338 (+ 244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.444899 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.445503 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.445670 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.445706 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.445805 (+ 99us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.445877 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.445935 (+ 58us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.445965 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251023 20:12:47.447726 18159 raft_consensus.cc:1275] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Refusing update from remote peer 1f49bf0f5f714183aa61273df0068c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.447726 18301 raft_consensus.cc:1275] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Refusing update from remote peer 1f49bf0f5f714183aa61273df0068c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.448038 18381 consensus_queue.cc:1048] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.448144 18385 consensus_queue.cc:1048] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.451053 18390 mvcc.cc:204] Tried to move back new op lower bound from 7214081505063383040 to 7214081505021284352. Current Snapshot: MvccSnapshot[applied={T|T < 7214081505063383040}]
I20251023 20:12:47.452265 18389 mvcc.cc:204] Tried to move back new op lower bound from 7214081505063383040 to 7214081505021284352. Current Snapshot: MvccSnapshot[applied={T|T < 7214081505063383040 or (T in {7214081505063383040})}]
I20251023 20:12:47.452625 17856 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:47.452749 17856 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:47.453480 17856 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: Sending DeleteTablet for 3 replicas of tablet 88b158cf055745899aa61268552e9d20
I20251023 20:12:47.453816 18139 tablet_service.cc:1552] Processing DeleteTablet for tablet 88b158cf055745899aa61268552e9d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:39298
I20251023 20:12:47.453833 18281 tablet_service.cc:1552] Processing DeleteTablet for tablet 88b158cf055745899aa61268552e9d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:46350
I20251023 20:12:47.453831 17997 tablet_service.cc:1552] Processing DeleteTablet for tablet 88b158cf055745899aa61268552e9d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:54752
I20251023 20:12:47.454128 18400 tablet_replica.cc:333] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: stopping tablet replica
I20251023 20:12:47.454213 18400 raft_consensus.cc:2243] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:12:47.454310 18400 raft_consensus.cc:2272] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.454353 17856 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:12:47.454510 17856 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:47.454762 18400 ts_tablet_manager.cc:1916] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.455693 17997 tablet_service.cc:1505] Processing CreateTablet for tablet 83d1db1d2f5344738b9f3d857d92077c (DEFAULT_TABLE table=test-table [id=8d838a44cb704129846a1283a314bd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.455761 18139 tablet_service.cc:1505] Processing CreateTablet for tablet 83d1db1d2f5344738b9f3d857d92077c (DEFAULT_TABLE table=test-table [id=8d838a44cb704129846a1283a314bd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.455813 17997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d1db1d2f5344738b9f3d857d92077c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.455857 18139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d1db1d2f5344738b9f3d857d92077c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.455919 18398 tablet_replica.cc:333] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: stopping tablet replica
I20251023 20:12:47.456005 18398 raft_consensus.cc:2243] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:47.456101 18398 raft_consensus.cc:2272] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.456967 18400 ts_tablet_manager.cc:1929] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:47.457026 18400 log.cc:1199] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/88b158cf055745899aa61268552e9d20
I20251023 20:12:47.457165 18376 tablet_bootstrap.cc:492] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: Bootstrap starting.
I20251023 20:12:47.457188 18375 tablet_bootstrap.cc:492] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: Bootstrap starting.
I20251023 20:12:47.457284 18400 ts_tablet_manager.cc:1950] T 88b158cf055745899aa61268552e9d20 P 1f49bf0f5f714183aa61273df0068c3f: Deleting consensus metadata
I20251023 20:12:47.457633 18376 tablet_bootstrap.cc:654] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.457706 18375 tablet_bootstrap.cc:654] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.457712 17844 catalog_manager.cc:4985] TS 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553): tablet 88b158cf055745899aa61268552e9d20 (table test-table [id=d12fd007f2294d37aa14683931b7b9eb]) successfully deleted
I20251023 20:12:47.458258 18376 tablet_bootstrap.cc:492] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: No bootstrap required, opened a new log
I20251023 20:12:47.458297 18376 ts_tablet_manager.cc:1403] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:47.458282 18281 tablet_service.cc:1505] Processing CreateTablet for tablet 83d1db1d2f5344738b9f3d857d92077c (DEFAULT_TABLE table=test-table [id=8d838a44cb704129846a1283a314bd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.458393 18281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d1db1d2f5344738b9f3d857d92077c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.458403 18375 tablet_bootstrap.cc:492] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: No bootstrap required, opened a new log
I20251023 20:12:47.458441 18375 ts_tablet_manager.cc:1403] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:47.458437 18376 raft_consensus.cc:359] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.458508 18376 raft_consensus.cc:385] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.458528 18376 raft_consensus.cc:740] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f49bf0f5f714183aa61273df0068c3f, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.458581 18376 consensus_queue.cc:260] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.458689 18376 ts_tablet_manager.cc:1434] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.458726 18375 raft_consensus.cc:359] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.458786 18375 raft_consensus.cc:385] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.458804 18375 raft_consensus.cc:740] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e934ee0612b44f4aaebb9063a6bb284e, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.458869 18375 consensus_queue.cc:260] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.458952 18375 ts_tablet_manager.cc:1434] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251023 20:12:47.459964 18377 tablet_bootstrap.cc:492] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: Bootstrap starting.
I20251023 20:12:47.460435 18377 tablet_bootstrap.cc:654] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.461063 18398 ts_tablet_manager.cc:1916] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.461225 18377 tablet_bootstrap.cc:492] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: No bootstrap required, opened a new log
I20251023 20:12:47.461282 18377 ts_tablet_manager.cc:1403] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:47.461439 18377 raft_consensus.cc:359] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.461516 18377 raft_consensus.cc:385] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.461539 18377 raft_consensus.cc:740] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9353b0ea583f4e069ce9ff52104c09ba, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.461603 18377 consensus_queue.cc:260] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.461702 18377 ts_tablet_manager.cc:1434] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.461982 18399 tablet_replica.cc:333] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: stopping tablet replica
I20251023 20:12:47.462074 18399 raft_consensus.cc:2243] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:47.462183 18399 raft_consensus.cc:2272] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.463222 18399 ts_tablet_manager.cc:1916] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.463218 18398 ts_tablet_manager.cc:1929] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:47.463287 18398 log.cc:1199] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/88b158cf055745899aa61268552e9d20
I20251023 20:12:47.463560 18398 ts_tablet_manager.cc:1950] T 88b158cf055745899aa61268552e9d20 P e934ee0612b44f4aaebb9063a6bb284e: Deleting consensus metadata
I20251023 20:12:47.464004 17845 catalog_manager.cc:4985] TS e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033): tablet 88b158cf055745899aa61268552e9d20 (table test-table [id=d12fd007f2294d37aa14683931b7b9eb]) successfully deleted
I20251023 20:12:47.464495 18399 ts_tablet_manager.cc:1929] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:47.464566 18399 log.cc:1199] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/88b158cf055745899aa61268552e9d20
I20251023 20:12:47.464841 18399 ts_tablet_manager.cc:1950] T 88b158cf055745899aa61268552e9d20 P 9353b0ea583f4e069ce9ff52104c09ba: Deleting consensus metadata
I20251023 20:12:47.465226 17843 catalog_manager.cc:4985] TS 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851): tablet 88b158cf055745899aa61268552e9d20 (table test-table [id=d12fd007f2294d37aa14683931b7b9eb]) successfully deleted
W20251023 20:12:47.556082 18348 tablet.cc:2378] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:12:47.609172 18383 raft_consensus.cc:493] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:47.609284 18383 raft_consensus.cc:515] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.609566 18383 leader_election.cc:290] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553), e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033)
I20251023 20:12:47.612584 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.609765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40033 (local address 127.9.117.195:59021)
1023 20:12:47.609831 (+ 66us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.609847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.609903 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.610126 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.610127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.610129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.610377 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.610379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.611324 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.611325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.612304 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.612306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.612409 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.612412 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.612446 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.612479 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251023 20:12:47.612699 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.609766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:38553 (local address 127.9.117.195:35031)
1023 20:12:47.609999 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.610010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.610072 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.610203 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.610204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.610205 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.610408 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.610410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.611324 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.611325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.612297 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.612301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.612387 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.612394 (+ 7us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.612432 (+ 38us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.612479 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":17,"thread_start_us":71,"threads_started":1}
I20251023 20:12:47.613106 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.609932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:59021 (local address 127.9.117.194:40033)
1023 20:12:47.609991 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.609993 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.610002 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:47.610033 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.610034 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.610042 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.610099 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.610462 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.611234 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.612433 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.612965 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.612974 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.613001 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.613035 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251023 20:12:47.613106 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.609890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:35031 (local address 127.9.117.193:38553)
1023 20:12:47.609957 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.609959 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.610083 (+ 124us) server_negotiation.cc:408] Connection header received
1023 20:12:47.610124 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.610125 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.610133 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.610188 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.610478 (+ 290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.611234 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.612418 (+ 1184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.612965 (+ 547us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.612974 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.613001 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.613037 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251023 20:12:47.613356 18159 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83d1db1d2f5344738b9f3d857d92077c" candidate_uuid: "9353b0ea583f4e069ce9ff52104c09ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e934ee0612b44f4aaebb9063a6bb284e" is_pre_election: true
I20251023 20:12:47.613374 18017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83d1db1d2f5344738b9f3d857d92077c" candidate_uuid: "9353b0ea583f4e069ce9ff52104c09ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f49bf0f5f714183aa61273df0068c3f" is_pre_election: true
I20251023 20:12:47.613451 18159 raft_consensus.cc:2468] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9353b0ea583f4e069ce9ff52104c09ba in term 0.
I20251023 20:12:47.613489 18017 raft_consensus.cc:2468] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9353b0ea583f4e069ce9ff52104c09ba in term 0.
I20251023 20:12:47.613672 18235 leader_election.cc:304] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9353b0ea583f4e069ce9ff52104c09ba, e934ee0612b44f4aaebb9063a6bb284e; no voters:
I20251023 20:12:47.613855 18383 raft_consensus.cc:2804] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:47.613895 18383 raft_consensus.cc:493] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:47.613921 18383 raft_consensus.cc:3060] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.614679 18383 raft_consensus.cc:515] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.614841 18383 leader_election.cc:290] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [CANDIDATE]: Term 1 election: Requested vote from peers 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553), e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033)
I20251023 20:12:47.615020 18017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83d1db1d2f5344738b9f3d857d92077c" candidate_uuid: "9353b0ea583f4e069ce9ff52104c09ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f49bf0f5f714183aa61273df0068c3f"
I20251023 20:12:47.615036 18159 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83d1db1d2f5344738b9f3d857d92077c" candidate_uuid: "9353b0ea583f4e069ce9ff52104c09ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e934ee0612b44f4aaebb9063a6bb284e"
I20251023 20:12:47.615097 18159 raft_consensus.cc:3060] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.615092 18017 raft_consensus.cc:3060] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.615741 18159 raft_consensus.cc:2468] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9353b0ea583f4e069ce9ff52104c09ba in term 1.
I20251023 20:12:47.615767 18017 raft_consensus.cc:2468] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9353b0ea583f4e069ce9ff52104c09ba in term 1.
I20251023 20:12:47.615931 18235 leader_election.cc:304] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9353b0ea583f4e069ce9ff52104c09ba, e934ee0612b44f4aaebb9063a6bb284e; no voters:
I20251023 20:12:47.616070 18383 raft_consensus.cc:2804] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:47.616216 18383 raft_consensus.cc:697] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 LEADER]: Becoming Leader. State: Replica: 9353b0ea583f4e069ce9ff52104c09ba, State: Running, Role: LEADER
I20251023 20:12:47.616286 18383 consensus_queue.cc:237] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.617053 17856 catalog_manager.cc:5649] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195). New cstate: current_term: 1 leader_uuid: "9353b0ea583f4e069ce9ff52104c09ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } health_report { overall_health: HEALTHY } } }
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250366, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.254@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:47.664861 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.657730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.9.117.254:43291)
1023 20:12:47.657799 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.657802 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.657933 (+ 131us) server_negotiation.cc:408] Connection header received
1023 20:12:47.657989 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.657990 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.657999 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.658051 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.658814 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.659303 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.659910 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.660032 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.662113 (+ 2081us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:47.662118 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:47.662119 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:47.662125 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:47.663693 (+ 1568us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.664113 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.664114 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.664114 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.664134 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.664321 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.664322 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.664322 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.664408 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:47.664457 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.664668 (+ 211us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.664790 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:12:47.665077 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.657661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43291 (local address 127.0.0.1:50174)
1023 20:12:47.657848 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.657859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.657941 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.658074 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.658074 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.658493 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:47.658697 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.658700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.659393 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.659394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.659827 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.659829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.659894 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.660449 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:47.660454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:47.662024 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:47.663796 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.663805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.663808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.664031 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.664224 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:47.664225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:47.664225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:47.664259 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:47.664494 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:47.664500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:47.664579 (+ 79us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.664648 (+ 69us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.664941 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":142,"thread_start_us":90,"threads_started":1}
I20251023 20:12:47.667156 17856 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:50174:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251023 20:12:47.667380 17856 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:47.669230 17997 tablet_service.cc:1505] Processing CreateTablet for tablet 7bec24f062fb4d769ed67b068a2c2887 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c507442e9a5e4db6b46d445997880858]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:12:47.669250 18281 tablet_service.cc:1505] Processing CreateTablet for tablet 7bec24f062fb4d769ed67b068a2c2887 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c507442e9a5e4db6b46d445997880858]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:12:47.669250 18139 tablet_service.cc:1505] Processing CreateTablet for tablet 7bec24f062fb4d769ed67b068a2c2887 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c507442e9a5e4db6b46d445997880858]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251023 20:12:47.669409 18281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bec24f062fb4d769ed67b068a2c2887. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.669409 18139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bec24f062fb4d769ed67b068a2c2887. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.669416 17997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bec24f062fb4d769ed67b068a2c2887. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.670572 18376 tablet_bootstrap.cc:492] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: Bootstrap starting.
I20251023 20:12:47.670935 18377 tablet_bootstrap.cc:492] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: Bootstrap starting.
I20251023 20:12:47.670960 18376 tablet_bootstrap.cc:654] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.671470 18377 tablet_bootstrap.cc:654] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.671540 18375 tablet_bootstrap.cc:492] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: Bootstrap starting.
I20251023 20:12:47.671658 18376 tablet_bootstrap.cc:492] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: No bootstrap required, opened a new log
I20251023 20:12:47.671710 18376 ts_tablet_manager.cc:1403] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:47.671834 18376 raft_consensus.cc:359] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.671901 18376 raft_consensus.cc:385] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.671931 18376 raft_consensus.cc:740] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f49bf0f5f714183aa61273df0068c3f, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.671977 18376 consensus_queue.cc:260] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.672048 18375 tablet_bootstrap.cc:654] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.672108 18376 ts_tablet_manager.cc:1434] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.672116 18394 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.672333 18394 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:12:47.672677 18377 tablet_bootstrap.cc:492] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: No bootstrap required, opened a new log
I20251023 20:12:47.672756 18377 ts_tablet_manager.cc:1403] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:12:47.672811 18375 tablet_bootstrap.cc:492] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: No bootstrap required, opened a new log
I20251023 20:12:47.672860 18375 ts_tablet_manager.cc:1403] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:47.672930 18377 raft_consensus.cc:359] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.672977 18377 raft_consensus.cc:385] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.672991 18377 raft_consensus.cc:740] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9353b0ea583f4e069ce9ff52104c09ba, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.673030 18377 consensus_queue.cc:260] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.673048 18375 raft_consensus.cc:359] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.673110 18375 raft_consensus.cc:385] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.673125 18375 raft_consensus.cc:740] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e934ee0612b44f4aaebb9063a6bb284e, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.673128 18377 ts_tablet_manager.cc:1434] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.673172 18375 consensus_queue.cc:260] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.673170 18383 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.673234 18383 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:12:47.673259 18375 ts_tablet_manager.cc:1434] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.673357 18382 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.673457 18382 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: This TxnStatusTablet replica's current role is: FOLLOWER
W20251023 20:12:47.708194 18064 tablet.cc:2378] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:12:47.714103 18382 raft_consensus.cc:493] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:47.714208 18382 raft_consensus.cc:515] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.714552 18382 leader_election.cc:290] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553), 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851)
I20251023 20:12:47.717020 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.714785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:38553 (local address 127.9.117.194:48575)
1023 20:12:47.714924 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.714934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.715008 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.715152 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.715153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.715155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.715412 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.715414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.716051 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.716052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.716717 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.716719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.716829 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.716834 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.716879 (+ 45us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.716924 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":58,"threads_started":1}
I20251023 20:12:47.717430 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.714851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:48575 (local address 127.9.117.193:38553)
1023 20:12:47.714904 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.714907 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.715002 (+ 95us) server_negotiation.cc:408] Connection header received
1023 20:12:47.715073 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.715074 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.715084 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.715143 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.715482 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.715970 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.716838 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.717310 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.717319 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.717341 (+ 22us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.717367 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:12:47.717584 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.714847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:43385 (local address 127.9.117.195:34851)
1023 20:12:47.714903 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.714906 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.714997 (+ 91us) server_negotiation.cc:408] Connection header received
1023 20:12:47.715072 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.715073 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.715085 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.715138 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:12:47.715486 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.715966 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.716854 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.717335 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.717344 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.717483 (+ 139us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.717535 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251023 20:12:47.717582 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.714759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34851 (local address 127.9.117.194:43385)
1023 20:12:47.714913 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.714925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.715005 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.715144 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.715144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.715146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:12:47.715413 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.715414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.716049 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.716051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.716712 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.716716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.717373 (+ 657us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.717376 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.717446 (+ 70us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.717489 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20251023 20:12:47.717662 18017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bec24f062fb4d769ed67b068a2c2887" candidate_uuid: "e934ee0612b44f4aaebb9063a6bb284e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f49bf0f5f714183aa61273df0068c3f" is_pre_election: true
I20251023 20:12:47.717762 18017 raft_consensus.cc:2468] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e934ee0612b44f4aaebb9063a6bb284e in term 0.
I20251023 20:12:47.717860 18301 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bec24f062fb4d769ed67b068a2c2887" candidate_uuid: "e934ee0612b44f4aaebb9063a6bb284e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9353b0ea583f4e069ce9ff52104c09ba" is_pre_election: true
I20251023 20:12:47.717953 18301 raft_consensus.cc:2468] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e934ee0612b44f4aaebb9063a6bb284e in term 0.
I20251023 20:12:47.717962 18092 leader_election.cc:304] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f49bf0f5f714183aa61273df0068c3f, e934ee0612b44f4aaebb9063a6bb284e; no voters:
I20251023 20:12:47.718103 18382 raft_consensus.cc:2804] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:47.718152 18382 raft_consensus.cc:493] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:47.718168 18382 raft_consensus.cc:3060] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.718809 18382 raft_consensus.cc:515] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.718971 18382 leader_election.cc:290] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [CANDIDATE]: Term 1 election: Requested vote from peers 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553), 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851)
I20251023 20:12:47.719141 18017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bec24f062fb4d769ed67b068a2c2887" candidate_uuid: "e934ee0612b44f4aaebb9063a6bb284e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f49bf0f5f714183aa61273df0068c3f"
I20251023 20:12:47.719205 18017 raft_consensus.cc:3060] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.719208 18301 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bec24f062fb4d769ed67b068a2c2887" candidate_uuid: "e934ee0612b44f4aaebb9063a6bb284e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9353b0ea583f4e069ce9ff52104c09ba"
I20251023 20:12:47.719285 18301 raft_consensus.cc:3060] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.719882 18017 raft_consensus.cc:2468] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e934ee0612b44f4aaebb9063a6bb284e in term 1.
I20251023 20:12:47.719949 18301 raft_consensus.cc:2468] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e934ee0612b44f4aaebb9063a6bb284e in term 1.
I20251023 20:12:47.720101 18092 leader_election.cc:304] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f49bf0f5f714183aa61273df0068c3f, e934ee0612b44f4aaebb9063a6bb284e; no voters:
I20251023 20:12:47.720206 18382 raft_consensus.cc:2804] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:47.720316 18382 raft_consensus.cc:697] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [term 1 LEADER]: Becoming Leader. State: Replica: e934ee0612b44f4aaebb9063a6bb284e, State: Running, Role: LEADER
I20251023 20:12:47.720373 18382 consensus_queue.cc:237] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.720779 18418 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: TxnStatusTablet state changed. Reason: New leader e934ee0612b44f4aaebb9063a6bb284e. Latest consensus state: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.720937 18418 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:12:47.721138 18420 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251023 20:12:47.721123 17856 catalog_manager.cc:5649] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e reported cstate change: term changed from 0 to 1, leader changed from <none> to e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194). New cstate: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } health_report { overall_health: UNKNOWN } } }
I20251023 20:12:47.721246 18420 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251023 20:12:47.733484 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.731295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.9.117.194:40033)
1023 20:12:47.731362 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.731365 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.731377 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:12:47.731447 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.731448 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.731470 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.731532 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:47.731826 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.732296 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.732969 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.733114 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.733144 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.733236 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.733304 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.733376 (+ 72us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.733401 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251023 20:12:47.733490 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.731218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40033 (local address 127.0.0.1:39312)
1023 20:12:47.731293 (+ 75us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.731303 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.731363 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.731537 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.731539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.731540 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:12:47.731730 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.731732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.732372 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.732373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.732870 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.732874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.732939 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.732964 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.733303 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.733315 (+ 12us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.733368 (+ 53us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.733425 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20251023 20:12:47.734736 18301 raft_consensus.cc:1275] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Refusing update from remote peer e934ee0612b44f4aaebb9063a6bb284e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.734930 18017 raft_consensus.cc:1275] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Refusing update from remote peer e934ee0612b44f4aaebb9063a6bb284e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.735051 18418 consensus_queue.cc:1048] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.735170 18382 consensus_queue.cc:1048] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.735888 18383 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: TxnStatusTablet state changed. Reason: New leader e934ee0612b44f4aaebb9063a6bb284e. Latest consensus state: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.735989 18383 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:12:47.737039 18382 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.737120 18382 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:12:47.737195 18383 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.737267 18383 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P 9353b0ea583f4e069ce9ff52104c09ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:12:47.738903 18394 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: TxnStatusTablet state changed. Reason: New leader e934ee0612b44f4aaebb9063a6bb284e. Latest consensus state: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.739010 18394 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:12:47.739421 18382 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.739511 18382 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P e934ee0612b44f4aaebb9063a6bb284e: This TxnStatusTablet replica's current role is: LEADER
I20251023 20:12:47.739813 18388 mvcc.cc:204] Tried to move back new op lower bound from 7214081506239459328 to 7214081506183286784. Current Snapshot: MvccSnapshot[applied={T|T < 7214081506239459328}]
I20251023 20:12:47.740029 18394 tablet_replica.cc:442] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } } }
I20251023 20:12:47.740105 18394 tablet_replica.cc:445] T 7bec24f062fb4d769ed67b068a2c2887 P 1f49bf0f5f714183aa61273df0068c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251023 20:12:47.741788 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.739159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.9.117.195:34851)
1023 20:12:47.739216 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.739218 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.739227 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:47.739271 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.739272 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.739290 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.739347 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:47.739704 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.740423 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.741317 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.741445 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.741482 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.741584 (+ 102us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.741654 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.741720 (+ 66us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.741740 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251023 20:12:47.746486 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.744086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40033 (local address 127.9.117.195:44915)
1023 20:12:47.744162 (+ 76us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.744172 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.744206 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.744346 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.744346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.744347 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:12:47.744541 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.744543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.745486 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.745487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.745995 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.745998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.746060 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.746077 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.746367 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.746368 (+ 1us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.746397 (+ 29us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.746429 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251023 20:12:47.746506 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.744165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:44915 (local address 127.9.117.194:40033)
1023 20:12:47.744220 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.744222 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.744232 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:47.744259 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.744260 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.744271 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.744340 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:47.744659 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.745394 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.746073 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.746252 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.746273 (+ 21us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.746301 (+ 28us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.746363 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.746412 (+ 49us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.746435 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251023 20:12:47.749186 18159 raft_consensus.cc:1275] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Refusing update from remote peer 9353b0ea583f4e069ce9ff52104c09ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.749200 18017 raft_consensus.cc:1275] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Refusing update from remote peer 9353b0ea583f4e069ce9ff52104c09ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.749428 18383 consensus_queue.cc:1048] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.749712 18405 consensus_queue.cc:1048] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.758057 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.755729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:51531 (local address 127.9.117.195:34851)
1023 20:12:47.755771 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.755774 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.755934 (+ 160us) server_negotiation.cc:408] Connection header received
1023 20:12:47.755995 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.755996 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.756007 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.756063 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:47.756344 (+ 281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.756869 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.757566 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.757734 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.757753 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.757775 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.757826 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.757955 (+ 129us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.757980 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251023 20:12:47.758073 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.755683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34851 (local address 127.9.117.194:51531)
1023 20:12:47.755862 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.755872 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:47.755934 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:47.756066 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:47.756067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:47.756069 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1023 20:12:47.756282 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.756284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.756949 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.756950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:47.757480 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:47.757483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.757551 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.757574 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.757855 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.757858 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:12:47.757932 (+ 74us) client_negotiation.cc:240] Negotiation successful
1023 20:12:47.757982 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20251023 20:12:47.769290 17857 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:47.769424 17857 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:47.770183 17857 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: Sending DeleteTablet for 3 replicas of tablet 83d1db1d2f5344738b9f3d857d92077c
I20251023 20:12:47.770495 17997 tablet_service.cc:1552] Processing DeleteTablet for tablet 83d1db1d2f5344738b9f3d857d92077c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:54752
I20251023 20:12:47.770643 18400 tablet_replica.cc:333] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: stopping tablet replica
I20251023 20:12:47.770687 18139 tablet_service.cc:1552] Processing DeleteTablet for tablet 83d1db1d2f5344738b9f3d857d92077c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:39298
I20251023 20:12:47.770723 18400 raft_consensus.cc:2243] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:47.770771 18400 raft_consensus.cc:2272] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.770797 18398 tablet_replica.cc:333] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: stopping tablet replica
I20251023 20:12:47.770841 18398 raft_consensus.cc:2243] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:47.771111 18281 tablet_service.cc:1552] Processing DeleteTablet for tablet 83d1db1d2f5344738b9f3d857d92077c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:46350
I20251023 20:12:47.771214 18399 tablet_replica.cc:333] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: stopping tablet replica
I20251023 20:12:47.771315 18399 raft_consensus.cc:2243] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:12:47.771415 18399 raft_consensus.cc:2272] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.771734 18400 ts_tablet_manager.cc:1916] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.773365 18400 ts_tablet_manager.cc:1929] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:12:47.773424 18400 log.cc:1199] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/83d1db1d2f5344738b9f3d857d92077c
I20251023 20:12:47.773631 18400 ts_tablet_manager.cc:1950] T 83d1db1d2f5344738b9f3d857d92077c P 1f49bf0f5f714183aa61273df0068c3f: Deleting consensus metadata
I20251023 20:12:47.773687 18399 ts_tablet_manager.cc:1916] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.773969 18398 raft_consensus.cc:2272] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.774367 18398 ts_tablet_manager.cc:1916] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.775493 18399 ts_tablet_manager.cc:1929] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:12:47.775560 18399 log.cc:1199] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/83d1db1d2f5344738b9f3d857d92077c
I20251023 20:12:47.775782 18399 ts_tablet_manager.cc:1950] T 83d1db1d2f5344738b9f3d857d92077c P 9353b0ea583f4e069ce9ff52104c09ba: Deleting consensus metadata
I20251023 20:12:47.776085 18398 ts_tablet_manager.cc:1929] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:12:47.776113 17844 catalog_manager.cc:4985] TS 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553): tablet 83d1db1d2f5344738b9f3d857d92077c (table test-table [id=8d838a44cb704129846a1283a314bd99]) successfully deleted
I20251023 20:12:47.776170 18398 log.cc:1199] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/83d1db1d2f5344738b9f3d857d92077c
I20251023 20:12:47.776221 17843 catalog_manager.cc:4985] TS 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851): tablet 83d1db1d2f5344738b9f3d857d92077c (table test-table [id=8d838a44cb704129846a1283a314bd99]) successfully deleted
I20251023 20:12:47.776376 18398 ts_tablet_manager.cc:1950] T 83d1db1d2f5344738b9f3d857d92077c P e934ee0612b44f4aaebb9063a6bb284e: Deleting consensus metadata
I20251023 20:12:47.776701 17845 catalog_manager.cc:4985] TS e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033): tablet 83d1db1d2f5344738b9f3d857d92077c (table test-table [id=8d838a44cb704129846a1283a314bd99]) successfully deleted
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
W20251023 20:12:47.795663 17918 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:47.790286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.9.117.254:43291)
1023 20:12:47.790346 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.790348 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.790494 (+ 146us) server_negotiation.cc:408] Connection header received
1023 20:12:47.790572 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.790574 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.790586 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.790644 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.791441 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.791950 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.792549 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.792797 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.795607 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50182: BlockingRecv error: recv got EOF from 127.0.0.1:50182 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Oct 23 20:12:47 dist-test-slave-x62p krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM
I20251023 20:12:47.803925 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.796894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.9.117.254:43291)
1023 20:12:47.796941 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.796944 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.797170 (+ 226us) server_negotiation.cc:408] Connection header received
1023 20:12:47.797239 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.797240 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.797252 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.797305 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:47.797939 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.798422 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.798960 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.799152 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.801399 (+ 2247us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:47.801405 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:47.801405 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:47.801413 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:47.802896 (+ 1483us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.803228 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.803230 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.803230 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.803253 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:47.803392 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:47.803392 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:47.803393 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:47.803502 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:47.803599 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.803710 (+ 111us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.803816 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:12:47.805614 17857 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:12:47.805804 17857 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:47.807341 18281 tablet_service.cc:1505] Processing CreateTablet for tablet 374675f8680542fbbc5bc9d03dcd2dc8 (DEFAULT_TABLE table=test-table [id=ba5693aecc574020b903906f0cdabcd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.807341 17997 tablet_service.cc:1505] Processing CreateTablet for tablet 374675f8680542fbbc5bc9d03dcd2dc8 (DEFAULT_TABLE table=test-table [id=ba5693aecc574020b903906f0cdabcd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.807489 17997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374675f8680542fbbc5bc9d03dcd2dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.807490 18281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374675f8680542fbbc5bc9d03dcd2dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.809060 18377 tablet_bootstrap.cc:492] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: Bootstrap starting.
I20251023 20:12:47.809553 18377 tablet_bootstrap.cc:654] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.809983 18139 tablet_service.cc:1505] Processing CreateTablet for tablet 374675f8680542fbbc5bc9d03dcd2dc8 (DEFAULT_TABLE table=test-table [id=ba5693aecc574020b903906f0cdabcd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.810122 18139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374675f8680542fbbc5bc9d03dcd2dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.811141 18375 tablet_bootstrap.cc:492] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: Bootstrap starting.
I20251023 20:12:47.811684 18377 tablet_bootstrap.cc:492] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: No bootstrap required, opened a new log
I20251023 20:12:47.811738 18375 tablet_bootstrap.cc:654] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.811741 18377 ts_tablet_manager.cc:1403] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:12:47.811973 18377 raft_consensus.cc:359] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.812055 18377 raft_consensus.cc:385] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.812081 18377 raft_consensus.cc:740] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9353b0ea583f4e069ce9ff52104c09ba, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.812130 18377 consensus_queue.cc:260] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.812227 18377 ts_tablet_manager.cc:1434] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:47.812749 18376 tablet_bootstrap.cc:492] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: Bootstrap starting.
I20251023 20:12:47.813151 18376 tablet_bootstrap.cc:654] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.813863 18376 tablet_bootstrap.cc:492] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: No bootstrap required, opened a new log
I20251023 20:12:47.813920 18376 ts_tablet_manager.cc:1403] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:47.814043 18376 raft_consensus.cc:359] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.814110 18376 raft_consensus.cc:385] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.814134 18376 raft_consensus.cc:740] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f49bf0f5f714183aa61273df0068c3f, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.814196 18376 consensus_queue.cc:260] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.814304 18376 ts_tablet_manager.cc:1434] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.814999 18375 tablet_bootstrap.cc:492] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: No bootstrap required, opened a new log
I20251023 20:12:47.815055 18375 ts_tablet_manager.cc:1403] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:12:47.815219 18375 raft_consensus.cc:359] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.815279 18375 raft_consensus.cc:385] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.815304 18375 raft_consensus.cc:740] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e934ee0612b44f4aaebb9063a6bb284e, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.815352 18375 consensus_queue.cc:260] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.815459 18375 ts_tablet_manager.cc:1434] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:47.846768 18418 raft_consensus.cc:493] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:47.846882 18418 raft_consensus.cc:515] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.847064 18418 leader_election.cc:290] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553), 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851)
I20251023 20:12:47.847268 18017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "374675f8680542fbbc5bc9d03dcd2dc8" candidate_uuid: "e934ee0612b44f4aaebb9063a6bb284e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f49bf0f5f714183aa61273df0068c3f" is_pre_election: true
I20251023 20:12:47.847302 18301 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "374675f8680542fbbc5bc9d03dcd2dc8" candidate_uuid: "e934ee0612b44f4aaebb9063a6bb284e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9353b0ea583f4e069ce9ff52104c09ba" is_pre_election: true
I20251023 20:12:47.847368 18017 raft_consensus.cc:2468] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e934ee0612b44f4aaebb9063a6bb284e in term 0.
I20251023 20:12:47.847380 18301 raft_consensus.cc:2468] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e934ee0612b44f4aaebb9063a6bb284e in term 0.
I20251023 20:12:47.847532 18092 leader_election.cc:304] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f49bf0f5f714183aa61273df0068c3f, e934ee0612b44f4aaebb9063a6bb284e; no voters:
I20251023 20:12:47.847661 18418 raft_consensus.cc:2804] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:47.847692 18418 raft_consensus.cc:493] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:47.847709 18418 raft_consensus.cc:3060] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.848448 18418 raft_consensus.cc:515] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.848618 18418 leader_election.cc:290] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [CANDIDATE]: Term 1 election: Requested vote from peers 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553), 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851)
I20251023 20:12:47.848868 18301 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "374675f8680542fbbc5bc9d03dcd2dc8" candidate_uuid: "e934ee0612b44f4aaebb9063a6bb284e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9353b0ea583f4e069ce9ff52104c09ba"
I20251023 20:12:47.848878 18017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "374675f8680542fbbc5bc9d03dcd2dc8" candidate_uuid: "e934ee0612b44f4aaebb9063a6bb284e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f49bf0f5f714183aa61273df0068c3f"
I20251023 20:12:47.848949 18017 raft_consensus.cc:3060] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.848949 18301 raft_consensus.cc:3060] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.849773 18301 raft_consensus.cc:2468] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e934ee0612b44f4aaebb9063a6bb284e in term 1.
I20251023 20:12:47.849844 18017 raft_consensus.cc:2468] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e934ee0612b44f4aaebb9063a6bb284e in term 1.
I20251023 20:12:47.849988 18091 leader_election.cc:304] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9353b0ea583f4e069ce9ff52104c09ba, e934ee0612b44f4aaebb9063a6bb284e; no voters:
I20251023 20:12:47.850198 18418 raft_consensus.cc:2804] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:47.850252 18418 raft_consensus.cc:697] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 1 LEADER]: Becoming Leader. State: Replica: e934ee0612b44f4aaebb9063a6bb284e, State: Running, Role: LEADER
I20251023 20:12:47.850302 18418 consensus_queue.cc:237] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.850857 17857 catalog_manager.cc:5649] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e reported cstate change: term changed from 0 to 1, leader changed from <none> to e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194). New cstate: current_term: 1 leader_uuid: "e934ee0612b44f4aaebb9063a6bb284e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } health_report { overall_health: UNKNOWN } } }
I20251023 20:12:47.864444 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.862362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.9.117.194:40033)
1023 20:12:47.862425 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.862427 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.862435 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:12:47.862464 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.862465 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.862475 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.862527 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:47.862794 (+ 267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.863254 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.863997 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.864149 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.864169 (+ 20us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.864191 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.864245 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.864318 (+ 73us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.864346 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251023 20:12:47.865504 18301 raft_consensus.cc:1275] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Refusing update from remote peer e934ee0612b44f4aaebb9063a6bb284e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.865516 18017 raft_consensus.cc:1275] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Refusing update from remote peer e934ee0612b44f4aaebb9063a6bb284e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.865752 18428 consensus_queue.cc:1048] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.866024 18428 consensus_queue.cc:1048] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.871001 17857 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50188:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:47.871110 17857 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50188:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:47.872023 17857 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: Sending DeleteTablet for 3 replicas of tablet 374675f8680542fbbc5bc9d03dcd2dc8
I20251023 20:12:47.872303 17997 tablet_service.cc:1552] Processing DeleteTablet for tablet 374675f8680542fbbc5bc9d03dcd2dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:54752
I20251023 20:12:47.872399 18400 tablet_replica.cc:333] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: stopping tablet replica
I20251023 20:12:47.872452 18400 raft_consensus.cc:2243] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:47.872502 18400 raft_consensus.cc:2272] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.872710 18139 tablet_service.cc:1552] Processing DeleteTablet for tablet 374675f8680542fbbc5bc9d03dcd2dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:39298
I20251023 20:12:47.872713 17858 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:12:47.872886 18398 tablet_replica.cc:333] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: stopping tablet replica
W20251023 20:12:47.872902 17858 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:12:47.872932 18398 raft_consensus.cc:2243] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:12:47.873024 18398 raft_consensus.cc:2272] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.872795 18138 tablet_service.cc:1552] Processing DeleteTablet for tablet 374675f8680542fbbc5bc9d03dcd2dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:39298
I20251023 20:12:47.873569 18281 tablet_service.cc:1552] Processing DeleteTablet for tablet 374675f8680542fbbc5bc9d03dcd2dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:46350
I20251023 20:12:47.873668 18399 tablet_replica.cc:333] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: stopping tablet replica
I20251023 20:12:47.873711 18399 raft_consensus.cc:2243] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:47.873759 18399 raft_consensus.cc:2272] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.873822 18398 ts_tablet_manager.cc:1916] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.874176 18399 ts_tablet_manager.cc:1916] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.875797 18399 ts_tablet_manager.cc:1929] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:47.875867 18399 log.cc:1199] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/374675f8680542fbbc5bc9d03dcd2dc8
I20251023 20:12:47.876085 18399 ts_tablet_manager.cc:1950] T 374675f8680542fbbc5bc9d03dcd2dc8 P 9353b0ea583f4e069ce9ff52104c09ba: Deleting consensus metadata
I20251023 20:12:47.876600 18398 ts_tablet_manager.cc:1929] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:47.876658 18398 log.cc:1199] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/374675f8680542fbbc5bc9d03dcd2dc8
I20251023 20:12:47.876897 18398 ts_tablet_manager.cc:1950] T 374675f8680542fbbc5bc9d03dcd2dc8 P e934ee0612b44f4aaebb9063a6bb284e: Deleting consensus metadata
I20251023 20:12:47.877565 18281 tablet_service.cc:1505] Processing CreateTablet for tablet 34480549fb0f4f429e38a00d5d93ab1c (DEFAULT_TABLE table=test-table [id=12353715a2c0459d8f0cc0c371a26800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.877581 17997 tablet_service.cc:1505] Processing CreateTablet for tablet 34480549fb0f4f429e38a00d5d93ab1c (DEFAULT_TABLE table=test-table [id=12353715a2c0459d8f0cc0c371a26800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.877692 17997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34480549fb0f4f429e38a00d5d93ab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.877691 18281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34480549fb0f4f429e38a00d5d93ab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.878931 18138 tablet_service.cc:1505] Processing CreateTablet for tablet 34480549fb0f4f429e38a00d5d93ab1c (DEFAULT_TABLE table=test-table [id=12353715a2c0459d8f0cc0c371a26800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:12:47.879047 18138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34480549fb0f4f429e38a00d5d93ab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:47.880331 18375 tablet_bootstrap.cc:492] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e: Bootstrap starting.
I20251023 20:12:47.880340 18377 tablet_bootstrap.cc:492] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: Bootstrap starting.
I20251023 20:12:47.880868 18375 tablet_bootstrap.cc:654] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.880869 18377 tablet_bootstrap.cc:654] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.881286 17845 catalog_manager.cc:4985] TS e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033): tablet 374675f8680542fbbc5bc9d03dcd2dc8 (table test-table [id=ba5693aecc574020b903906f0cdabcd3]) successfully deleted
W20251023 20:12:47.881419 17845 catalog_manager.cc:4952] TS e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033): delete failed for tablet 374675f8680542fbbc5bc9d03dcd2dc8 because the tablet was not found. No further retry: Not found: Tablet not found: 374675f8680542fbbc5bc9d03dcd2dc8
I20251023 20:12:47.882040 18376 tablet_bootstrap.cc:492] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: Bootstrap starting.
I20251023 20:12:47.882426 18376 tablet_bootstrap.cc:654] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:47.882676 18400 ts_tablet_manager.cc:1916] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.882848 17843 catalog_manager.cc:4985] TS 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851): tablet 374675f8680542fbbc5bc9d03dcd2dc8 (table test-table [id=ba5693aecc574020b903906f0cdabcd3]) successfully deleted
I20251023 20:12:47.883351 18376 tablet_bootstrap.cc:492] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: No bootstrap required, opened a new log
I20251023 20:12:47.883400 18376 ts_tablet_manager.cc:1403] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:47.883533 18376 raft_consensus.cc:359] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.883607 18376 raft_consensus.cc:385] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.883632 18376 raft_consensus.cc:740] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f49bf0f5f714183aa61273df0068c3f, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.883684 18376 consensus_queue.cc:260] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.883770 18376 ts_tablet_manager.cc:1434] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.884235 18400 ts_tablet_manager.cc:1929] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:12:47.884287 18400 log.cc:1199] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/374675f8680542fbbc5bc9d03dcd2dc8
I20251023 20:12:47.884507 18400 ts_tablet_manager.cc:1950] T 374675f8680542fbbc5bc9d03dcd2dc8 P 1f49bf0f5f714183aa61273df0068c3f: Deleting consensus metadata
I20251023 20:12:47.884966 17844 catalog_manager.cc:4985] TS 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553): tablet 374675f8680542fbbc5bc9d03dcd2dc8 (table test-table [id=ba5693aecc574020b903906f0cdabcd3]) successfully deleted
I20251023 20:12:47.885495 18377 tablet_bootstrap.cc:492] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: No bootstrap required, opened a new log
I20251023 20:12:47.885556 18377 ts_tablet_manager.cc:1403] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251023 20:12:47.885715 18375 tablet_bootstrap.cc:492] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e: No bootstrap required, opened a new log
I20251023 20:12:47.885759 18375 ts_tablet_manager.cc:1403] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20251023 20:12:47.885717 18377 raft_consensus.cc:359] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.885790 18377 raft_consensus.cc:385] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.885833 18377 raft_consensus.cc:740] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9353b0ea583f4e069ce9ff52104c09ba, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.885878 18377 consensus_queue.cc:260] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.885908 18375 raft_consensus.cc:359] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.885967 18375 raft_consensus.cc:385] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:47.885988 18375 raft_consensus.cc:740] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e934ee0612b44f4aaebb9063a6bb284e, State: Initialized, Role: FOLLOWER
I20251023 20:12:47.885982 18377 ts_tablet_manager.cc:1434] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.886034 18375 consensus_queue.cc:260] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.886122 18375 ts_tablet_manager.cc:1434] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:47.914338 18383 raft_consensus.cc:493] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:12:47.914475 18383 raft_consensus.cc:515] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.914657 18383 leader_election.cc:290] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553), e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033)
I20251023 20:12:47.914932 18017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34480549fb0f4f429e38a00d5d93ab1c" candidate_uuid: "9353b0ea583f4e069ce9ff52104c09ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f49bf0f5f714183aa61273df0068c3f" is_pre_election: true
I20251023 20:12:47.914933 18159 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34480549fb0f4f429e38a00d5d93ab1c" candidate_uuid: "9353b0ea583f4e069ce9ff52104c09ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e934ee0612b44f4aaebb9063a6bb284e" is_pre_election: true
I20251023 20:12:47.915032 18017 raft_consensus.cc:2468] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9353b0ea583f4e069ce9ff52104c09ba in term 0.
I20251023 20:12:47.915045 18159 raft_consensus.cc:2468] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9353b0ea583f4e069ce9ff52104c09ba in term 0.
I20251023 20:12:47.915207 18234 leader_election.cc:304] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f49bf0f5f714183aa61273df0068c3f, 9353b0ea583f4e069ce9ff52104c09ba; no voters:
I20251023 20:12:47.915297 18383 raft_consensus.cc:2804] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:12:47.915323 18383 raft_consensus.cc:493] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:12:47.915342 18383 raft_consensus.cc:3060] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.916266 18383 raft_consensus.cc:515] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.916417 18383 leader_election.cc:290] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [CANDIDATE]: Term 1 election: Requested vote from peers 1f49bf0f5f714183aa61273df0068c3f (127.9.117.193:38553), e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033)
I20251023 20:12:47.916642 18159 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34480549fb0f4f429e38a00d5d93ab1c" candidate_uuid: "9353b0ea583f4e069ce9ff52104c09ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e934ee0612b44f4aaebb9063a6bb284e"
I20251023 20:12:47.916649 18017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34480549fb0f4f429e38a00d5d93ab1c" candidate_uuid: "9353b0ea583f4e069ce9ff52104c09ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f49bf0f5f714183aa61273df0068c3f"
I20251023 20:12:47.916750 18017 raft_consensus.cc:3060] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.916752 18159 raft_consensus.cc:3060] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:47.917659 18017 raft_consensus.cc:2468] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9353b0ea583f4e069ce9ff52104c09ba in term 1.
I20251023 20:12:47.917714 18159 raft_consensus.cc:2468] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9353b0ea583f4e069ce9ff52104c09ba in term 1.
I20251023 20:12:47.917857 18234 leader_election.cc:304] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f49bf0f5f714183aa61273df0068c3f, 9353b0ea583f4e069ce9ff52104c09ba; no voters:
I20251023 20:12:47.917944 18383 raft_consensus.cc:2804] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:47.917984 18383 raft_consensus.cc:697] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 LEADER]: Becoming Leader. State: Replica: 9353b0ea583f4e069ce9ff52104c09ba, State: Running, Role: LEADER
I20251023 20:12:47.918027 18383 consensus_queue.cc:237] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } }
I20251023 20:12:47.918648 17858 catalog_manager.cc:5649] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195). New cstate: current_term: 1 leader_uuid: "9353b0ea583f4e069ce9ff52104c09ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9353b0ea583f4e069ce9ff52104c09ba" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34851 } health_report { overall_health: HEALTHY } } }
I20251023 20:12:47.935901 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.933339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.9.117.195:34851)
1023 20:12:47.933411 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:47.933414 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:47.933426 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:12:47.933451 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:47.933453 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:47.933465 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:47.933534 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:12:47.933860 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.934559 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:47.935362 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:47.935539 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:47.935564 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:12:47.935593 (+ 29us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:12:47.935657 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:47.935780 (+ 123us) server_negotiation.cc:300] Negotiation successful
1023 20:12:47.935808 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:12:47.938623 18017 raft_consensus.cc:1275] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Refusing update from remote peer 9353b0ea583f4e069ce9ff52104c09ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.938840 18383 consensus_queue.cc:1048] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f49bf0f5f714183aa61273df0068c3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.939678 18159 raft_consensus.cc:1275] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Refusing update from remote peer 9353b0ea583f4e069ce9ff52104c09ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:12:47.939931 18383 consensus_queue.cc:1048] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e934ee0612b44f4aaebb9063a6bb284e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:12:47.954578 17858 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50188:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:47.954711 17858 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50188:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:12:47.955659 17858 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 16bc966345ee4ab9a689d4ea11589e35: Sending DeleteTablet for 3 replicas of tablet 34480549fb0f4f429e38a00d5d93ab1c
I20251023 20:12:47.955979 17997 tablet_service.cc:1552] Processing DeleteTablet for tablet 34480549fb0f4f429e38a00d5d93ab1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:54752
I20251023 20:12:47.956086 18400 tablet_replica.cc:333] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: stopping tablet replica
I20251023 20:12:47.956140 18400 raft_consensus.cc:2243] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:47.956188 18400 raft_consensus.cc:2272] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.956537 18400 ts_tablet_manager.cc:1916] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.957141 18138 tablet_service.cc:1552] Processing DeleteTablet for tablet 34480549fb0f4f429e38a00d5d93ab1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:39298
I20251023 20:12:47.957240 18398 tablet_replica.cc:333] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e: stopping tablet replica
I20251023 20:12:47.957288 18398 raft_consensus.cc:2243] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:12:47.957325 18398 raft_consensus.cc:2272] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.957921 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 17932
I20251023 20:12:47.958290 18400 ts_tablet_manager.cc:1929] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:12:47.958343 18400 log.cc:1199] T 34480549fb0f4f429e38a00d5d93ab1c P 1f49bf0f5f714183aa61273df0068c3f: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/34480549fb0f4f429e38a00d5d93ab1c
I20251023 20:12:47.965103 18281 tablet_service.cc:1552] Processing DeleteTablet for tablet 34480549fb0f4f429e38a00d5d93ab1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:12:47 UTC) from {username='oryx', principal='oryx/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:46350
I20251023 20:12:47.965252 18399 tablet_replica.cc:333] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: stopping tablet replica
I20251023 20:12:47.965302 18399 raft_consensus.cc:2243] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:12:47.965395 18399 raft_consensus.cc:2272] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:12:47.965833 18399 ts_tablet_manager.cc:1916] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.967660 18399 ts_tablet_manager.cc:1929] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251023 20:12:47.967736 18399 log.cc:1199] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/34480549fb0f4f429e38a00d5d93ab1c
I20251023 20:12:47.967953 18399 ts_tablet_manager.cc:1950] T 34480549fb0f4f429e38a00d5d93ab1c P 9353b0ea583f4e069ce9ff52104c09ba: Deleting consensus metadata
I20251023 20:12:47.968413 17843 catalog_manager.cc:4985] TS 9353b0ea583f4e069ce9ff52104c09ba (127.9.117.195:34851): tablet 34480549fb0f4f429e38a00d5d93ab1c (table test-table [id=12353715a2c0459d8f0cc0c371a26800]) successfully deleted
W20251023 20:12:47.969000 17844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:38553 (error 108)
I20251023 20:12:47.969009 18398 ts_tablet_manager.cc:1916] T 34480549fb0f4f429e38a00d5d93ab1c P e934ee0612b44f4aaebb9063a6bb284e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:12:47.969794 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 18074
I20251023 20:12:47.976294 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.973481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:38553 (local address 127.0.0.1:54764)
1023 20:12:47.976188 (+ 2707us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.976234 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:38553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2685,"thread_start_us":63,"threads_started":1}
I20251023 20:12:47.976321 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 18216
I20251023 20:12:47.976424 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:47.976278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40033 (local address 127.0.0.1:39330)
1023 20:12:47.976382 (+ 104us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:47.976403 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.194:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
W20251023 20:12:47.977834 17845 catalog_manager.cc:4712] TS e934ee0612b44f4aaebb9063a6bb284e (127.9.117.194:40033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34480549fb0f4f429e38a00d5d93ab1c: Network error: Client connection negotiation failed: client connection to 127.9.117.194:40033: connect: Connection refused (error 111)
I20251023 20:12:47.982561 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 17826
2025-10-23T20:12:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3517 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:48 dist-test-slave-x62p krb5kdc[18466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:48 dist-test-slave-x62p krb5kdc[18466](info): set up 2 sockets
Oct 23 20:12:48 dist-test-slave-x62p krb5kdc[18466](info): commencing operation
krb5kdc: starting...
W20251023 20:12:49.973671 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:50 dist-test-slave-x62p krb5kdc[18466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:50.111403 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40623
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:50.197690 18482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:50.197896 18482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:50.197921 18482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:50.199402 18482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:50.199463 18482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:50.199479 18482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:50.199491 18482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:50.199503 18482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:50.201323 18482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:40623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:50.201604 18482 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:50.201823 18482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:50.204799 18487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:50.204869 18488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:50.204800 18490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:50.205221 18482 server_base.cc:1047] running on GCE node
I20251023 20:12:50.205446 18482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:50.205730 18482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:50.206873 18482 hybrid_clock.cc:648] HybridClock initialized: now 1761250370206851 us; error 44 us; skew 500 ppm
W20251023 20:12:50.208648 18482 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-10-23T20:12:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2225 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:12:50 dist-test-slave-x62p krb5kdc[18495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:12:50 dist-test-slave-x62p krb5kdc[18495](info): set up 2 sockets
Oct 23 20:12:50 dist-test-slave-x62p krb5kdc[18495](info): commencing operation
krb5kdc: starting...
W20251023 20:12:52.205667 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.976s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:12:52 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:12:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:12:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:52.337652 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33491
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:52.419135 18511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:52.419319 18511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:52.419358 18511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:52.420809 18511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:52.420858 18511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:52.420882 18511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:52.420905 18511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:52.420924 18511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:52.422520 18511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:52.422819 18511 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:52.423017 18511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:52.425577 18517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:52.425580 18519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:52.425580 18516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:52.425839 18511 server_base.cc:1047] running on GCE node
I20251023 20:12:52.426048 18511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:52.426292 18511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:52.427440 18511 hybrid_clock.cc:648] HybridClock initialized: now 1761250372427421 us; error 29 us; skew 500 ppm
Oct 23 20:12:52 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:52.429756 18511 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:52.430322 18511 webserver.cc:492] Webserver started at http://127.9.117.254:46143/ using document root <none> and password file <none>
I20251023 20:12:52.430497 18511 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:52.430528 18511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:52.430632 18511 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:52.431602 18511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "4ccebb55d39148319f3a6892664b70d2"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "4443534ccd5afac0a7cfeac4cf00daea"
server_key_iv: "4dfe4547aa5d77d7e7ff58d6cc9a14ce"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.431898 18511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "4ccebb55d39148319f3a6892664b70d2"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "4443534ccd5afac0a7cfeac4cf00daea"
server_key_iv: "4dfe4547aa5d77d7e7ff58d6cc9a14ce"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.433210 18511 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:12:52.433889 18526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:52.434087 18511 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:12:52.434161 18511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "4ccebb55d39148319f3a6892664b70d2"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "4443534ccd5afac0a7cfeac4cf00daea"
server_key_iv: "4dfe4547aa5d77d7e7ff58d6cc9a14ce"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.434217 18511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:52.455495 18511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:52.457468 18511 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:52.457580 18511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:52.461901 18511 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:33491
I20251023 20:12:52.461977 18578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:33491 every 8 connection(s)
I20251023 20:12:52.462265 18511 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:52.462931 18579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:12:52.465600 18579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Bootstrap starting.
I20251023 20:12:52.466183 18579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Neither blocks nor log segments found. Creating new log.
I20251023 20:12:52.466408 18579 log.cc:826] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:52.467159 18579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: No bootstrap required, opened a new log
I20251023 20:12:52.468369 18579 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } }
I20251023 20:12:52.468493 18579 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:12:52.468514 18579 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccebb55d39148319f3a6892664b70d2, State: Initialized, Role: FOLLOWER
I20251023 20:12:52.468621 18579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } }
I20251023 20:12:52.468672 18579 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:52.468708 18579 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:52.468787 18579 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:12:52.469352 18579 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } }
I20251023 20:12:52.469460 18579 leader_election.cc:304] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ccebb55d39148319f3a6892664b70d2; no voters:
I20251023 20:12:52.469636 18579 leader_election.cc:290] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:12:52.469695 18582 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:12:52.469870 18579 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:52.469873 18582 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4ccebb55d39148319f3a6892664b70d2, State: Running, Role: LEADER
I20251023 20:12:52.470093 18582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } }
I20251023 20:12:52.470412 18583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ccebb55d39148319f3a6892664b70d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } } }
I20251023 20:12:52.470448 18584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ccebb55d39148319f3a6892664b70d2. Latest consensus state: current_term: 1 leader_uuid: "4ccebb55d39148319f3a6892664b70d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } } }
I20251023 20:12:52.470564 18583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:52.470570 18584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:52.470811 18590 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:52.471513 18590 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:52.472466 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 18511
I20251023 20:12:52.472610 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:12:52.472828 9687 external_mini_cluster.cc:1468] Setting key 6e697966e770d0ea8de5c0eee52af0c0
I20251023 20:12:52.473107 18590 catalog_manager.cc:1357] Generated new cluster ID: a1454f8e478040e285ff295e53d35b5c
I20251023 20:12:52.473158 18590 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:12:52 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:52.483206 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:52.473282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.9.117.254:33491)
1023 20:12:52.473586 (+ 304us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:52.473600 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:52.473621 (+ 21us) server_negotiation.cc:408] Connection header received
1023 20:12:52.473739 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:52.473745 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:52.473794 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:52.473945 (+ 151us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:52.474861 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.476050 (+ 1189us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:52.476792 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.477044 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:52.479631 (+ 2587us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:52.479637 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:52.479638 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:52.479661 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:52.481715 (+ 2054us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:52.482124 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:52.482127 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:52.482128 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:52.482190 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:52.482350 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:52.482350 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:52.482351 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:52.482599 (+ 248us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:52.482723 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:52.482898 (+ 175us) server_negotiation.cc:300] Negotiation successful
1023 20:12:52.483020 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":99,"threads_started":1}
I20251023 20:12:52.496057 18590 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:12:52.496887 18590 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:52.504248 18590 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Generated new TSK 0
I20251023 20:12:52.504460 18590 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:52.555478 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:33491
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:52.640722 18607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:52.640959 18607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:52.640985 18607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:52.642403 18607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:52.642444 18607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:52.642472 18607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:52.644021 18607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:52.644228 18607 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:52.644383 18607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:12:52.647655 18607 server_base.cc:1047] running on GCE node
W20251023 20:12:52.647498 18615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:52.647642 18612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:52.647498 18613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:52.648190 18607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:52.648461 18607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:52.649636 18607 hybrid_clock.cc:648] HybridClock initialized: now 1761250372649620 us; error 34 us; skew 500 ppm
Oct 23 20:12:52 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:52.652232 18607 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:12:52.652896 18607 webserver.cc:492] Webserver started at http://127.9.117.193:45359/ using document root <none> and password file <none>
I20251023 20:12:52.653119 18607 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:52.653162 18607 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:52.653290 18607 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:52.654328 18607 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "ab2e89d0be8f452abc2fd664cbec1e55"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "c2842a6f858e88188e3ddb4b342e7966"
server_key_iv: "a8cdefe6e74d285d4258e2e01bb866f2"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.654644 18607 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "ab2e89d0be8f452abc2fd664cbec1e55"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "c2842a6f858e88188e3ddb4b342e7966"
server_key_iv: "a8cdefe6e74d285d4258e2e01bb866f2"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.655896 18607 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:12:52.656531 18622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:52.656713 18607 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:52.656821 18607 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "ab2e89d0be8f452abc2fd664cbec1e55"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "c2842a6f858e88188e3ddb4b342e7966"
server_key_iv: "a8cdefe6e74d285d4258e2e01bb866f2"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.656898 18607 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:52.679672 18607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:52.681658 18607 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:52.681803 18607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:52.682049 18607 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:52.682389 18607 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:52.682425 18607 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:52.682463 18607 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:52.682494 18607 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:52.688529 18607 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:41985
I20251023 20:12:52.688581 18735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:41985 every 8 connection(s)
I20251023 20:12:52.688926 18607 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:52.690409 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 18607
I20251023 20:12:52.690542 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:12:52.690728 9687 external_mini_cluster.cc:1468] Setting key e8ae0045afa4a232a417f1611e04534c
Oct 23 20:12:52 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:52.698865 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:52.689574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:34793)
1023 20:12:52.689707 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:52.689735 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:52.690231 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:52.690510 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:52.690514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:52.690917 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:52.691253 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:52.691257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.692183 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:52.692184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:52.693244 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:52.693247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.693414 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:52.694041 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:52.694062 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:52.695741 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:52.697698 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:52.697704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:52.697707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:52.697919 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:52.698088 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:52.698088 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:52.698088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:52.698141 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:52.698375 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:52.698378 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:52.698497 (+ 119us) client_negotiation.cc:769] Sending connection context
1023 20:12:52.698581 (+ 84us) client_negotiation.cc:240] Negotiation successful
1023 20:12:52.698708 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
I20251023 20:12:52.698870 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:52.689619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:34793 (local address 127.9.117.254:33491)
1023 20:12:52.689725 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:52.689727 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:52.689818 (+ 91us) server_negotiation.cc:408] Connection header received
1023 20:12:52.690344 (+ 526us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:52.690346 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:52.690367 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:52.690457 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:52.691334 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.692094 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:52.693336 (+ 1242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.693470 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:52.695840 (+ 2370us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:52.695845 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:52.695845 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:52.695855 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:52.697607 (+ 1752us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:52.697991 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:52.697993 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:52.697993 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:52.698021 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:52.698191 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:52.698192 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:52.698193 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:52.698319 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:52.698392 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:52.698625 (+ 233us) server_negotiation.cc:300] Negotiation successful
1023 20:12:52.698746 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251023 20:12:52.699410 18736 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33491
I20251023 20:12:52.699507 18736 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:52.699717 18736 heartbeater.cc:507] Master 127.9.117.254:33491 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:12:52.700237 18543 ts_manager.cc:194] Registered new tserver with Master: ab2e89d0be8f452abc2fd664cbec1e55 (127.9.117.193:41985)
I20251023 20:12:52.700959 18543 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:34793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:52.739645 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:33491
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:52.821336 18743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:52.821537 18743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:52.821563 18743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:52.823051 18743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:52.823105 18743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:52.823133 18743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:52.824782 18743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:52.825053 18743 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:52.825243 18743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:52.828084 18748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:52.828147 18749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:52.828084 18751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:52.828284 18743 server_base.cc:1047] running on GCE node
I20251023 20:12:52.828454 18743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:52.828658 18743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:52.829799 18743 hybrid_clock.cc:648] HybridClock initialized: now 1761250372829801 us; error 50 us; skew 500 ppm
Oct 23 20:12:52 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:52.832252 18743 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:12:52.832880 18743 webserver.cc:492] Webserver started at http://127.9.117.194:41933/ using document root <none> and password file <none>
I20251023 20:12:52.833060 18743 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:52.833091 18743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:52.833185 18743 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:52.834133 18743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "949cc21d3bb24806be07da59f910ddee"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "80374909f6ab1379f14fdb108d188abc"
server_key_iv: "5cdfed9db8bc7b943872f8b7867dc782"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.834407 18743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "949cc21d3bb24806be07da59f910ddee"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "80374909f6ab1379f14fdb108d188abc"
server_key_iv: "5cdfed9db8bc7b943872f8b7867dc782"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.835562 18743 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:52.836262 18758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:52.836467 18743 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:52.836650 18743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "949cc21d3bb24806be07da59f910ddee"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "80374909f6ab1379f14fdb108d188abc"
server_key_iv: "5cdfed9db8bc7b943872f8b7867dc782"
server_key_version: "encryptionkey@0"
I20251023 20:12:52.836761 18743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:52.856457 18743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:52.858423 18743 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:52.858556 18743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:52.858752 18743 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:52.859059 18743 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:52.859091 18743 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:52.859114 18743 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:52.859126 18743 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:52.865321 18743 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:32905
I20251023 20:12:52.865373 18871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:32905 every 8 connection(s)
I20251023 20:12:52.865696 18743 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:12:52 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:52.874318 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:52.866384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:58067 (local address 127.9.117.254:33491)
1023 20:12:52.866457 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:52.866460 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:52.866661 (+ 201us) server_negotiation.cc:408] Connection header received
1023 20:12:52.867178 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:52.867180 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:52.867191 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:52.867258 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:52.868070 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.868579 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:52.869699 (+ 1120us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.869842 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:52.871927 (+ 2085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:52.871933 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:52.871934 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:52.871941 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:52.873281 (+ 1340us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:52.873644 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:52.873645 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:52.873645 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:52.873663 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:52.873802 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:52.873802 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:52.873803 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:52.873902 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:52.873949 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:52.874175 (+ 226us) server_negotiation.cc:300] Negotiation successful
1023 20:12:52.874246 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:12:52.874459 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:52.866347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:58067)
1023 20:12:52.866525 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:52.866556 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:52.867066 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:52.867312 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:52.867316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:52.867661 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:52.867968 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:52.867972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.868710 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:52.868712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:52.869594 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:52.869597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:52.869731 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:52.870338 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:52.870362 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:52.871821 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:52.873359 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:52.873365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:52.873368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:52.873578 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:52.873708 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:52.873708 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:52.873709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:52.873762 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:52.873949 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:52.873953 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:52.874075 (+ 122us) client_negotiation.cc:769] Sending connection context
1023 20:12:52.874163 (+ 88us) client_negotiation.cc:240] Negotiation successful
1023 20:12:52.874274 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20251023 20:12:52.874594 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 18743
I20251023 20:12:52.874684 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:12:52.874876 9687 external_mini_cluster.cc:1468] Setting key aa1d6323dc813953db65f13aa732a096
I20251023 20:12:52.874970 18872 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33491
I20251023 20:12:52.875095 18872 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:52.875320 18872 heartbeater.cc:507] Master 127.9.117.254:33491 requested a full tablet report, sending...
I20251023 20:12:52.875733 18543 ts_manager.cc:194] Registered new tserver with Master: 949cc21d3bb24806be07da59f910ddee (127.9.117.194:32905)
I20251023 20:12:52.876264 18543 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:58067
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:12:52.926077 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:33491
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:53.009562 18879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:53.009757 18879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:53.009799 18879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:53.011265 18879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:53.011317 18879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:53.011361 18879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:53.013044 18879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:53.013317 18879 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:53.013510 18879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:53.016276 18887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:53.016276 18884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:53.016357 18879 server_base.cc:1047] running on GCE node
W20251023 20:12:53.016286 18885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:53.016683 18879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:53.016911 18879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:53.018062 18879 hybrid_clock.cc:648] HybridClock initialized: now 1761250373018034 us; error 47 us; skew 500 ppm
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250373, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:53.020691 18879 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:12:53.021277 18879 webserver.cc:492] Webserver started at http://127.9.117.195:44617/ using document root <none> and password file <none>
I20251023 20:12:53.021469 18879 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:53.021510 18879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:53.021651 18879 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:12:53.022658 18879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "21c36a62252b4a5c99875b35b0bec9ca"
format_stamp: "Formatted at 2025-10-23 20:12:53 on dist-test-slave-x62p"
server_key: "7f849a52f5113aa76c22c6443b852789"
server_key_iv: "82e7a00d759a28560516231475b0c88d"
server_key_version: "encryptionkey@0"
I20251023 20:12:53.022971 18879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "21c36a62252b4a5c99875b35b0bec9ca"
format_stamp: "Formatted at 2025-10-23 20:12:53 on dist-test-slave-x62p"
server_key: "7f849a52f5113aa76c22c6443b852789"
server_key_iv: "82e7a00d759a28560516231475b0c88d"
server_key_version: "encryptionkey@0"
I20251023 20:12:53.024190 18879 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251023 20:12:53.025019 18894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.025207 18879 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:53.025295 18879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "21c36a62252b4a5c99875b35b0bec9ca"
format_stamp: "Formatted at 2025-10-23 20:12:53 on dist-test-slave-x62p"
server_key: "7f849a52f5113aa76c22c6443b852789"
server_key_iv: "82e7a00d759a28560516231475b0c88d"
server_key_version: "encryptionkey@0"
I20251023 20:12:53.025363 18879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:53.036157 18879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:53.038533 18879 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:53.038703 18879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:53.038957 18879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:53.039309 18879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:53.039345 18879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.039381 18879 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:53.039410 18879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.045603 18879 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:33729
I20251023 20:12:53.045651 19007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:33729 every 8 connection(s)
I20251023 20:12:53.046015 18879 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:12:53.050961 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 18879
I20251023 20:12:53.051069 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:12:53.051239 9687 external_mini_cluster.cc:1468] Setting key 55aeb078df3b108d4608ec6e11af0da3
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250373, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:12:53.055510 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.046738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:45795 (local address 127.9.117.254:33491)
1023 20:12:53.046828 (+ 90us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.046831 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.046996 (+ 165us) server_negotiation.cc:408] Connection header received
1023 20:12:53.047513 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.047514 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.047525 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.047583 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.048406 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.048994 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.050074 (+ 1080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.050200 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.052531 (+ 2331us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:53.052534 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:53.052534 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:53.052541 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:53.054036 (+ 1495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:53.054484 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:53.054484 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:53.054485 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:53.054502 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:53.054685 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:53.054685 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:53.054685 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:53.054783 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:53.054831 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:53.055345 (+ 514us) server_negotiation.cc:300] Negotiation successful
1023 20:12:53.055433 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251023 20:12:53.055740 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.046706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:45795)
1023 20:12:53.046880 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.046909 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.047417 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.047648 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.047652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.048005 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.048307 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.048310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.049102 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.049103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.049985 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.049987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.050110 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.050691 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.050711 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.052449 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:12:53.054167 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:53.054175 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:53.054177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:53.054414 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:53.054550 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:12:53.054550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:12:53.054550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:12:53.054641 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:12:53.054870 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:12:53.054876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:12:53.055029 (+ 153us) client_negotiation.cc:769] Sending connection context
1023 20:12:53.055325 (+ 296us) client_negotiation.cc:240] Negotiation successful
1023 20:12:53.055482 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20251023 20:12:53.056246 19008 heartbeater.cc:344] Connected to a master server at 127.9.117.254:33491
I20251023 20:12:53.056365 19008 heartbeater.cc:461] Registering TS with master...
I20251023 20:12:53.056622 19008 heartbeater.cc:507] Master 127.9.117.254:33491 requested a full tablet report, sending...
I20251023 20:12:53.057039 18543 ts_manager.cc:194] Registered new tserver with Master: 21c36a62252b4a5c99875b35b0bec9ca (127.9.117.195:33729)
I20251023 20:12:53.057443 18543 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:45795
I20251023 20:12:53.063055 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.193.keytab.
Entry for principal oryx/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.193.keytab.
WARNING: no policy specified for oryx/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.194.keytab.
Entry for principal oryx/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.194.keytab.
WARNING: no policy specified for oryx/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.195.keytab.
Entry for principal oryx/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.195.keytab.
I20251023 20:12:53.135932 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 18607
I20251023 20:12:53.142865 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 18743
I20251023 20:12:53.147964 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 18879
I20251023 20:12:53.152280 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 18511
I20251023 20:12:53.157603 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33491
--webserver_interface=127.9.117.254
--webserver_port=46143
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:12:53.238969 19017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:53.239136 19017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:53.239158 19017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:53.240556 19017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:12:53.240623 19017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:53.240638 19017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:53.240651 19017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:12:53.240664 19017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:12:53.242225 19017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=46143
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:53.242486 19017 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:53.242676 19017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:53.245111 19023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:53.245191 19022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:53.245256 19017 server_base.cc:1047] running on GCE node
W20251023 20:12:53.245117 19025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:53.245540 19017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:53.245792 19017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:53.246939 19017 hybrid_clock.cc:648] HybridClock initialized: now 1761250373246929 us; error 29 us; skew 500 ppm
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250373, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:53.249249 19017 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:12:53.249775 19017 webserver.cc:492] Webserver started at http://127.9.117.254:46143/ using document root <none> and password file <none>
I20251023 20:12:53.249962 19017 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:53.250016 19017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:53.251257 19017 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:53.251892 19032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.252070 19017 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:12:53.252142 19017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "4ccebb55d39148319f3a6892664b70d2"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "4443534ccd5afac0a7cfeac4cf00daea"
server_key_iv: "4dfe4547aa5d77d7e7ff58d6cc9a14ce"
server_key_version: "encryptionkey@0"
I20251023 20:12:53.252435 19017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:53.272301 19017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:53.274459 19017 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:53.274613 19017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:53.278725 19017 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:33491
I20251023 20:12:53.278795 19084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:33491 every 8 connection(s)
I20251023 20:12:53.279174 19017 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:12:53.280711 19085 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:12:53.281533 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Bootstrap starting.
I20251023 20:12:53.282100 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 19017
I20251023 20:12:53.282399 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:41985
--local_ip_for_outbound_sockets=127.9.117.193
--tserver_master_addrs=127.9.117.254:33491
--webserver_port=45359
--webserver_interface=127.9.117.193
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.117.193
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:12:53.290907 19085 log.cc:826] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:12:53.292637 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:12:53.292946 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Bootstrap complete.
I20251023 20:12:53.295038 19085 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } }
I20251023 20:12:53.295224 19085 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccebb55d39148319f3a6892664b70d2, State: Initialized, Role: FOLLOWER
I20251023 20:12:53.295384 19085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } }
I20251023 20:12:53.295455 19085 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:12:53.295487 19085 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:12:53.295532 19085 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:12:53.296355 19085 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } }
I20251023 20:12:53.296447 19085 leader_election.cc:304] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ccebb55d39148319f3a6892664b70d2; no voters:
I20251023 20:12:53.296622 19085 leader_election.cc:290] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251023 20:12:53.296675 19089 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:12:53.296823 19089 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [term 2 LEADER]: Becoming Leader. State: Replica: 4ccebb55d39148319f3a6892664b70d2, State: Running, Role: LEADER
I20251023 20:12:53.296885 19085 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:12:53.296912 19089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } }
I20251023 20:12:53.297211 19090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ccebb55d39148319f3a6892664b70d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } } }
I20251023 20:12:53.297297 19090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:53.297204 19089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ccebb55d39148319f3a6892664b70d2. Latest consensus state: current_term: 2 leader_uuid: "4ccebb55d39148319f3a6892664b70d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccebb55d39148319f3a6892664b70d2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33491 } } }
I20251023 20:12:53.297498 19089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:12:53.297748 19094 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:12:53.298252 19094 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:12:53.298400 19094 catalog_manager.cc:1269] Loaded cluster ID: a1454f8e478040e285ff295e53d35b5c
I20251023 20:12:53.298480 19094 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:12:53.299978 19094 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:12:53.300148 19094 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 4ccebb55d39148319f3a6892664b70d2: Loaded TSK: 0
I20251023 20:12:53.300338 19094 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251023 20:12:53.366843 19087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:53.367053 19087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:53.367098 19087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:53.368639 19087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:53.368700 19087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:53.368770 19087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:12:53.370465 19087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.193.keytab
--principal=oryx/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:41985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=45359
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:53.370754 19087 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:53.370963 19087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:53.373706 19111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:53.373719 19112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:53.373965 19087 server_base.cc:1047] running on GCE node
W20251023 20:12:53.374074 19114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:53.374271 19087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:53.374548 19087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:53.375728 19087 hybrid_clock.cc:648] HybridClock initialized: now 1761250373375716 us; error 36 us; skew 500 ppm
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250373, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:53.378608 19087 init.cc:377] Logged in from keytab as oryx/127.9.117.193@KRBTEST.COM (short username oryx)
I20251023 20:12:53.379303 19087 webserver.cc:492] Webserver started at http://127.9.117.193:45359/ using document root <none> and password file <none>
I20251023 20:12:53.379536 19087 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:53.379616 19087 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:53.381281 19087 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:12:53.381920 19121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.382097 19087 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:53.382164 19087 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "ab2e89d0be8f452abc2fd664cbec1e55"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "c2842a6f858e88188e3ddb4b342e7966"
server_key_iv: "a8cdefe6e74d285d4258e2e01bb866f2"
server_key_version: "encryptionkey@0"
I20251023 20:12:53.382468 19087 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:53.401890 19087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:53.403983 19087 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:53.404101 19087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:53.404349 19087 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:53.404698 19087 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:53.404773 19087 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.404819 19087 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:53.404848 19087 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.411432 19087 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:41985
I20251023 20:12:53.411484 19234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:41985 every 8 connection(s)
I20251023 20:12:53.411834 19087 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:12:53.417330 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 19087
I20251023 20:12:53.417695 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:32905
--local_ip_for_outbound_sockets=127.9.117.194
--tserver_master_addrs=127.9.117.254:33491
--webserver_port=41933
--webserver_interface=127.9.117.194
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.117.194
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:53.421568 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.412501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:38921)
1023 20:12:53.412684 (+ 183us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.412720 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.413358 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.413825 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.413829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.414250 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.414675 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.414680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.415737 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.415738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.417226 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.417230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.417509 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.418364 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.418393 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.421375 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
W20251023 20:12:53.421764 19238 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.412616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:38921 (local address 127.9.117.254:33491)
1023 20:12:53.412808 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.412817 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.412840 (+ 23us) server_negotiation.cc:408] Connection header received
1023 20:12:53.413603 (+ 763us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.413608 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.413650 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.413761 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.414806 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.415640 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.417358 (+ 1718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.417660 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.421578 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:38921: BlockingRecv error: recv got EOF from 127.9.117.193:38921 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
W20251023 20:12:53.421824 19235 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:33491 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.117.254:33491: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:53.427613 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.422064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:45421)
1023 20:12:53.422119 (+ 55us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.422131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.422176 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.422328 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.422329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.422504 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.422741 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.422744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.423745 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.423746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.424429 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.424432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.424519 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.425270 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.425275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.427518 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20251023 20:12:53.427696 19238 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.422185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:45421 (local address 127.9.117.254:33491)
1023 20:12:53.422222 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.422224 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.422231 (+ 7us) server_negotiation.cc:408] Connection header received
1023 20:12:53.422258 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.422258 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.422268 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.422317 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.422844 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.423637 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.424527 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.424718 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.427634 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:45421: BlockingRecv error: recv got EOF from 127.9.117.193:45421 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:53.433615 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.427913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:39131)
1023 20:12:53.428802 (+ 889us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.428816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.428887 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.429101 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.429102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.429253 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.429459 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.429462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.430378 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.430379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.430989 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.430992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.431057 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.431579 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.431584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.433522 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":868}
W20251023 20:12:53.433753 19235 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251023 20:12:53.433915 19238 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.428018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:39131 (local address 127.9.117.254:33491)
1023 20:12:53.428061 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.428064 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.428888 (+ 824us) server_negotiation.cc:408] Connection header received
1023 20:12:53.428989 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.428990 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.429007 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.429085 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.429556 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.430278 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.431084 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.431649 (+ 565us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.433639 (+ 1990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:39131: BlockingRecv error: recv got EOF from 127.9.117.193:39131 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20251023 20:12:53.503023 19239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:53.503208 19239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:53.503245 19239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:53.504693 19239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:53.504763 19239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:53.504812 19239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:12:53.506389 19239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.194.keytab
--principal=oryx/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:32905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=41933
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:53.506678 19239 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:53.506876 19239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:53.509949 19247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:53.509938 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:53.509938 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:53.510161 19239 server_base.cc:1047] running on GCE node
I20251023 20:12:53.510376 19239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:53.510563 19239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:53.511693 19239 hybrid_clock.cc:648] HybridClock initialized: now 1761250373511677 us; error 34 us; skew 500 ppm
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250373, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:53.515161 19239 init.cc:377] Logged in from keytab as oryx/127.9.117.194@KRBTEST.COM (short username oryx)
I20251023 20:12:53.515795 19239 webserver.cc:492] Webserver started at http://127.9.117.194:41933/ using document root <none> and password file <none>
I20251023 20:12:53.515982 19239 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:53.516026 19239 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:53.517537 19239 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:53.518214 19254 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.518440 19239 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:12:53.518515 19239 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "949cc21d3bb24806be07da59f910ddee"
format_stamp: "Formatted at 2025-10-23 20:12:52 on dist-test-slave-x62p"
server_key: "80374909f6ab1379f14fdb108d188abc"
server_key_iv: "5cdfed9db8bc7b943872f8b7867dc782"
server_key_version: "encryptionkey@0"
I20251023 20:12:53.518867 19239 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:53.534385 19239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:53.536177 19239 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:53.536295 19239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:53.536526 19239 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:53.536906 19239 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:53.536943 19239 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.536978 19239 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:53.537004 19239 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.543150 19239 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:32905
I20251023 20:12:53.543246 19367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:32905 every 8 connection(s)
I20251023 20:12:53.543556 19239 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
W20251023 20:12:53.550143 19238 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.544344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:40625 (local address 127.9.117.254:33491)
1023 20:12:53.544434 (+ 90us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.544436 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.544551 (+ 115us) server_negotiation.cc:408] Connection header received
1023 20:12:53.545067 (+ 516us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.545068 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.545083 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.545136 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.545868 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.546340 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.547313 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.547427 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.550090 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:40625: BlockingRecv error: recv got EOF from 127.9.117.194:40625 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20251023 20:12:53.550158 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.544294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:40625)
1023 20:12:53.544437 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.544465 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.544983 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.545182 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.545186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.545497 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.545792 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.545796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.546404 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.546405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.547241 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.547244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.547355 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.547851 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.547866 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.549964 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
W20251023 20:12:53.550426 19368 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:33491 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.117.254:33491: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
I20251023 20:12:53.552954 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 19239
I20251023 20:12:53.553227 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:33729
--local_ip_for_outbound_sockets=127.9.117.195
--tserver_master_addrs=127.9.117.254:33491
--webserver_port=44617
--webserver_interface=127.9.117.195
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.117.195
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:53.554477 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.550645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:54613)
1023 20:12:53.550695 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.550705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.550748 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.550891 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.550892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.551017 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.551159 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.551161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.551750 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.551751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.552107 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.552109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.552173 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.552668 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.552672 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.554417 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20251023 20:12:53.554584 19238 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.550712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:54613 (local address 127.9.117.254:33491)
1023 20:12:53.550774 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.550776 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.550786 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:53.550812 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.550813 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.550825 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.550871 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.551229 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.551678 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.552179 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.552351 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.554516 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:54613: BlockingRecv error: recv got EOF from 127.9.117.194:54613 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:53.559057 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.554818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:54363)
1023 20:12:53.554870 (+ 52us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.554893 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.554939 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.555071 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.555072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.555187 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.555301 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.555303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.555928 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.555929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.556468 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.556470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.556545 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.557070 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.557075 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.558980 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20251023 20:12:53.559193 19238 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.554916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:54363 (local address 127.9.117.254:33491)
1023 20:12:53.554953 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.554956 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.554963 (+ 7us) server_negotiation.cc:408] Connection header received
1023 20:12:53.554988 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.554988 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.554998 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.555051 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.555372 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.555855 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.556592 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.556796 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.559074 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:54363: BlockingRecv error: recv got EOF from 127.9.117.194:54363 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20251023 20:12:53.559206 19368 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251023 20:12:53.638881 19371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:12:53.639065 19371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:12:53.639106 19371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:12:53.640551 19371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:12:53.640611 19371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:12:53.640657 19371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:12:53.642223 19371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/krb5kdc/oryx_127.9.117.195.keytab
--principal=oryx/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:33729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=44617
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:12:53.642498 19371 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:12:53.642699 19371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:12:53.645375 19376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:53.645433 19377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:12:53.645371 19379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:12:53.645391 19371 server_base.cc:1047] running on GCE node
I20251023 20:12:53.645709 19371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:12:53.645912 19371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:12:53.647042 19371 hybrid_clock.cc:648] HybridClock initialized: now 1761250373647028 us; error 32 us; skew 500 ppm
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250373, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:12:53.649263 19371 init.cc:377] Logged in from keytab as oryx/127.9.117.195@KRBTEST.COM (short username oryx)
I20251023 20:12:53.649809 19371 webserver.cc:492] Webserver started at http://127.9.117.195:44617/ using document root <none> and password file <none>
I20251023 20:12:53.650002 19371 fs_manager.cc:362] Metadata directory not provided
I20251023 20:12:53.650058 19371 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:12:53.651340 19371 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:12:53.651859 19386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.652052 19371 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.652123 19371 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "21c36a62252b4a5c99875b35b0bec9ca"
format_stamp: "Formatted at 2025-10-23 20:12:53 on dist-test-slave-x62p"
server_key: "7f849a52f5113aa76c22c6443b852789"
server_key_iv: "82e7a00d759a28560516231475b0c88d"
server_key_version: "encryptionkey@0"
I20251023 20:12:53.652417 19371 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:12:53.664314 19371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:12:53.666205 19371 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:12:53.666345 19371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:12:53.666558 19371 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:12:53.666874 19371 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:12:53.666908 19371 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.666944 19371 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:12:53.666973 19371 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:12:53.672715 19371 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:33729
I20251023 20:12:53.672816 19499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:33729 every 8 connection(s)
I20251023 20:12:53.673115 19371 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:53.679463 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 19371
W20251023 20:12:53.679654 19238 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.673809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:50909 (local address 127.9.117.254:33491)
1023 20:12:53.673883 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.673886 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.674025 (+ 139us) server_negotiation.cc:408] Connection header received
1023 20:12:53.674533 (+ 508us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.674534 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.674548 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.674601 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.675375 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.675856 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.676882 (+ 1026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.677061 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.679609 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:50909: BlockingRecv error: recv got EOF from 127.9.117.195:50909 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20251023 20:12:53.679687 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.673770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:50909)
1023 20:12:53.673917 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.673943 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.674426 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.674673 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.674677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.674982 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.675258 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.675261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.675938 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.675939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.676788 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.676790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.676921 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.677445 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.677460 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.679512 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
W20251023 20:12:53.679891 19500 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:33491 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.117.254:33491: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:53.684772 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.680099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:60351)
1023 20:12:53.680132 (+ 33us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.680142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.680182 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.680408 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.680409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.680552 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.680804 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.680814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.681593 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.681594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.682041 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.682043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.682106 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.682823 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.682827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.684675 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20251023 20:12:53.684898 19504 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.680154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:60351 (local address 127.9.117.254:33491)
1023 20:12:53.680289 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.680291 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.680298 (+ 7us) server_negotiation.cc:408] Connection header received
1023 20:12:53.680314 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.680315 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.680323 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.680387 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.680901 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.681517 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.682260 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.682435 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.684794 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:60351: BlockingRecv error: recv got EOF from 127.9.117.195:60351 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20251023 20:12:53.687237 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.680059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.9.117.254:33491)
1023 20:12:53.680175 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.680177 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.680185 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:12:53.680206 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.680207 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.680236 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.680313 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.681215 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.681780 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.682572 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.682744 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.683764 (+ 1020us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:12:53.683773 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:12:53.683773 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:12:53.683799 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:12:53.685854 (+ 2055us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:53.686253 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:53.686258 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:53.686258 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:53.686315 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:12:53.686475 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:12:53.686476 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:12:53.686476 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:12:53.686690 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:12:53.686828 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1023 20:12:53.686985 (+ 157us) server_negotiation.cc:300] Negotiation successful
1023 20:12:53.687136 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:53 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:53.689015 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:53.685107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:50139)
1023 20:12:53.685158 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:53.685170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:53.685216 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:53.685352 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:53.685353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:53.685475 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:53.685596 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.685597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.686384 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.686385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:53.686723 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:53.686724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.686785 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.687201 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:53.687204 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:53.688970 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20251023 20:12:53.689131 19504 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:53.685190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:50139 (local address 127.9.117.254:33491)
1023 20:12:53.685243 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:53.685245 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:53.685253 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:12:53.685276 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:53.685277 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:53.685288 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:53.685347 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:53.685667 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.686307 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:53.686789 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:53.686972 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:53.689066 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:50139: BlockingRecv error: recv got EOF from 127.9.117.195:50139 (error 108)
Metrics: {"server-negotiator.queue_time_us":37,"thread_start_us":104,"threads_started":1}
W20251023 20:12:53.689168 19500 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Oct 23 20:12:54 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:54 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:54.438937 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:54.434278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:33667)
1023 20:12:54.434477 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:54.434492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:54.434581 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:54.434732 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:54.434733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:54.434928 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:54.435110 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:54.435112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.435846 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:54.435846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:54.436268 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:54.436269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.436329 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:54.436961 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:54.436965 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:54.438837 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
W20251023 20:12:54.439052 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:54.434413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:33667 (local address 127.9.117.254:33491)
1023 20:12:54.434584 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:54.434586 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:54.434617 (+ 31us) server_negotiation.cc:408] Connection header received
1023 20:12:54.434646 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:54.434647 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:54.434658 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:54.434722 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:54.435214 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.435758 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:54.436350 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.436479 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:54.438968 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:33667: BlockingRecv error: recv got EOF from 127.9.117.193:33667 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
Oct 23 20:12:54 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:54 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:54.563977 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:54.559676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:43879)
1023 20:12:54.559822 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:54.559836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:54.559913 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:54.560066 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:54.560067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:54.560250 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:54.560437 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:54.560440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.561121 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:54.561122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:54.561542 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:54.561544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.561611 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:54.562129 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:54.562133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:54.563899 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
W20251023 20:12:54.564088 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:54.559729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:43879 (local address 127.9.117.254:33491)
1023 20:12:54.559802 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:54.559804 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:54.559915 (+ 111us) server_negotiation.cc:408] Connection header received
1023 20:12:54.559972 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:54.559973 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:54.559988 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:54.560052 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:54.560515 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.561037 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:54.561632 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.561753 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:54.564019 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:43879: BlockingRecv error: recv got EOF from 127.9.117.194:43879 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Oct 23 20:12:54 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:54 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:54.694279 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:54.689742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:32943)
1023 20:12:54.689903 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:54.689915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:54.689978 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:54.690148 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:54.690149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:54.690314 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:54.690499 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:54.690501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.691371 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:54.691372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:54.691781 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:54.691783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.691842 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:54.692408 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:54.692413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:54.694211 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
W20251023 20:12:54.694373 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:54.689848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:32943 (local address 127.9.117.254:33491)
1023 20:12:54.689914 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:54.689916 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:54.689973 (+ 57us) server_negotiation.cc:408] Connection header received
1023 20:12:54.690032 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:54.690033 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:54.690045 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:54.690125 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:54.690585 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.691293 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:54.691845 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:54.691955 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:54.694308 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:32943: BlockingRecv error: recv got EOF from 127.9.117.195:32943 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 23 20:12:55 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:55 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:55.447407 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:55.439542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:52243)
1023 20:12:55.439690 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:55.439704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:55.439775 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:55.439940 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:55.439941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:55.440159 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:55.440366 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:55.440369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.441048 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:55.441049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:55.441403 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:55.441404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.441459 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:55.445525 (+ 4066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:55.445530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:55.447344 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
W20251023 20:12:55.447539 19511 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:55.439656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:52243 (local address 127.9.117.254:33491)
1023 20:12:55.439800 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:55.439802 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:55.439811 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:55.439834 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:55.439835 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:55.439844 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:55.439917 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:55.440435 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.440980 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:55.441461 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.441603 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:55.447462 (+ 5859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:52243: BlockingRecv error: recv got EOF from 127.9.117.193:52243 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
Oct 23 20:12:55 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:55 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:55.569149 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:55.564592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:50277)
1023 20:12:55.564792 (+ 200us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:55.564803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:55.564866 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:55.565029 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:55.565030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:55.565213 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:55.565381 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:55.565383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.566063 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:55.566065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:55.566441 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:55.566442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.566512 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:55.567087 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:55.567091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:55.569069 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
W20251023 20:12:55.569267 19511 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:55.564813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:50277 (local address 127.9.117.254:33491)
1023 20:12:55.564896 (+ 83us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:55.564898 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:55.564909 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:12:55.564940 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:55.564941 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:55.564953 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:55.565009 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:55.565474 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.565976 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:55.566523 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.566739 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:55.569200 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:50277: BlockingRecv error: recv got EOF from 127.9.117.194:50277 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Oct 23 20:12:55 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:55 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:55.699030 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:55.694738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:51561)
1023 20:12:55.694918 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:55.694930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:55.695034 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:55.695173 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:55.695174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:55.695347 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:55.695517 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:55.695518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.696157 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:55.696158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:55.696539 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:55.696541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.696603 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:55.697159 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:55.697164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:55.698963 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
W20251023 20:12:55.699165 19511 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:55.694910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:51561 (local address 127.9.117.254:33491)
1023 20:12:55.694975 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:55.694977 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:55.695009 (+ 32us) server_negotiation.cc:408] Connection header received
1023 20:12:55.695098 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:55.695099 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:55.695109 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:55.695157 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:55.695583 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.696088 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:55.696608 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:55.696713 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:55.699081 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:51561: BlockingRecv error: recv got EOF from 127.9.117.195:51561 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 23 20:12:56 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:56 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:56.452527 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:56.447937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:46957)
1023 20:12:56.448080 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:56.448090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:56.448146 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:56.448326 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:56.448327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:56.448500 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:56.448673 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:56.448675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.449437 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:56.449439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:56.449833 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:56.449835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.449909 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:56.450467 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:56.450472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:56.452444 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
W20251023 20:12:56.452656 19515 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:56.448034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:46957 (local address 127.9.117.254:33491)
1023 20:12:56.448177 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:56.448185 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:56.448198 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:12:56.448229 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:56.448229 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:56.448241 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:56.448317 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:56.448804 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.449343 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:56.449942 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.450144 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:56.452584 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:46957: BlockingRecv error: recv got EOF from 127.9.117.193:46957 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
Oct 23 20:12:56 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:56 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:56.574023 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:56.569825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:47005)
1023 20:12:56.569974 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:56.569984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:56.570053 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:56.570227 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:56.570229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:56.570403 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:56.570575 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:56.570577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.571259 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:56.571260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:56.571654 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:56.571655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.571715 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:56.572227 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:56.572231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:56.573963 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
W20251023 20:12:56.574134 19515 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:56.569905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:47005 (local address 127.9.117.254:33491)
1023 20:12:56.569986 (+ 81us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:56.569988 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:56.570049 (+ 61us) server_negotiation.cc:408] Connection header received
1023 20:12:56.570110 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:56.570111 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:56.570123 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:56.570183 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:56.570691 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.571194 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:56.571723 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.571839 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:56.574069 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:47005: BlockingRecv error: recv got EOF from 127.9.117.194:47005 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 23 20:12:56 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:56 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:56.703547 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:56.699503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:56507)
1023 20:12:56.699678 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:56.699689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:56.699758 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:56.699893 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:56.699894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:56.700064 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:56.700227 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:56.700228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.700876 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:56.700877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:56.701263 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:56.701265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.701325 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:56.701826 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:56.701829 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:56.703483 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
W20251023 20:12:56.703683 19515 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:56.699572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:56507 (local address 127.9.117.254:33491)
1023 20:12:56.699635 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:56.699637 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:56.699751 (+ 114us) server_negotiation.cc:408] Connection header received
1023 20:12:56.699806 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:56.699807 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:56.699815 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:56.699878 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:56.700283 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.700795 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:56.701325 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:56.701436 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:56.703609 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:56507: BlockingRecv error: recv got EOF from 127.9.117.195:56507 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Oct 23 20:12:57 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:57 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:57.457250 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:57.453146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:55037)
1023 20:12:57.453283 (+ 137us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:57.453294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:57.453354 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:57.453483 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:57.453484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:57.453653 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:57.453824 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:57.453826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.454466 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:57.454467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:57.454875 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:57.454886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.454946 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:57.455444 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:57.455448 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:57.457190 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
W20251023 20:12:57.457408 19519 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:57.453216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:55037 (local address 127.9.117.254:33491)
1023 20:12:57.453353 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:57.453356 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:57.453365 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:12:57.453393 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:57.453394 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:57.453405 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:57.453473 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:57.453886 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.454393 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:57.454953 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.455086 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:57.457310 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:55037: BlockingRecv error: recv got EOF from 127.9.117.193:55037 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
Oct 23 20:12:57 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:57 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:57.579269 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:57.574641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:47265)
1023 20:12:57.574816 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:57.574831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:57.574913 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:57.575071 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:57.575072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:57.575267 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:57.575437 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:57.575439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.576125 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:57.576127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:57.576530 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:57.576532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.576613 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:57.577279 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:57.577284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:57.579205 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
W20251023 20:12:57.579389 19519 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:57.574767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:47265 (local address 127.9.117.254:33491)
1023 20:12:57.574857 (+ 90us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:57.574860 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:57.574888 (+ 28us) server_negotiation.cc:408] Connection header received
1023 20:12:57.574986 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:57.574987 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:57.575003 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:57.575056 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:57.575528 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.576044 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:57.576620 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.576873 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:57.579322 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:47265: BlockingRecv error: recv got EOF from 127.9.117.194:47265 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 23 20:12:57 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:57 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:57.708914 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:57.704164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:49841)
1023 20:12:57.704364 (+ 200us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:57.704376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:57.704466 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:57.704663 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:57.704664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:57.705056 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:57.705298 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:57.705302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.706046 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:57.706047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:57.706425 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:57.706427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.706485 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:57.707009 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:57.707013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:57.708810 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
W20251023 20:12:57.709017 19519 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:57.704290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:49841 (local address 127.9.117.254:33491)
1023 20:12:57.704353 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:57.704355 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:57.704446 (+ 91us) server_negotiation.cc:408] Connection header received
1023 20:12:57.704539 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:57.704540 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:57.704555 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:57.704641 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:57.705438 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.705967 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:57.706497 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:57.706608 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:57.708948 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:49841: BlockingRecv error: recv got EOF from 127.9.117.195:49841 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Oct 23 20:12:58 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:58 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:58.462152 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:58.457757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:48577)
1023 20:12:58.457940 (+ 183us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:58.457952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:58.458016 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:58.458166 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:58.458167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:58.458337 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:58.458507 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:58.458509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.459216 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:58.459217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:58.459632 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:58.459634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.459694 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:58.460237 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:58.460241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:58.462091 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
W20251023 20:12:58.462276 19523 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:58.457873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:48577 (local address 127.9.117.254:33491)
1023 20:12:58.457989 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:58.457991 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:58.458001 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:12:58.458075 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:58.458076 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:58.458086 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:58.458148 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:58.458613 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.459133 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:58.459708 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.459839 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:58.462200 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:48577: BlockingRecv error: recv got EOF from 127.9.117.193:48577 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
Oct 23 20:12:58 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:58 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:58.584197 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:58.579901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:52391)
1023 20:12:58.580102 (+ 201us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:58.580118 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:58.580192 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:58.580344 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:58.580345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:58.580546 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:58.580764 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:58.580766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.581424 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:58.581425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:58.581808 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:58.581809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.581865 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:58.582390 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:58.582394 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:58.584135 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
W20251023 20:12:58.584311 19523 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:58.580106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:52391 (local address 127.9.117.254:33491)
1023 20:12:58.580206 (+ 100us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:58.580208 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:58.580239 (+ 31us) server_negotiation.cc:408] Connection header received
1023 20:12:58.580266 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:58.580267 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:58.580278 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:58.580325 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:58.580852 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.581357 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:58.581898 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.582077 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:58.584244 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:52391: BlockingRecv error: recv got EOF from 127.9.117.194:52391 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Oct 23 20:12:58 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:58 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:58.714273 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:58.709505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:59807)
1023 20:12:58.709668 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:58.709685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:58.709779 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:58.709962 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:58.709963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:58.710182 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:58.710346 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:58.710348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.711022 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:58.711023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:58.711404 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:58.711406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.711473 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:58.712040 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:58.712045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:58.714208 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
W20251023 20:12:58.714421 19523 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:58.709632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:59807 (local address 127.9.117.254:33491)
1023 20:12:58.709695 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:58.709697 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:58.709762 (+ 65us) server_negotiation.cc:408] Connection header received
1023 20:12:58.709857 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:58.709858 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:58.709871 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:58.709925 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:58.710436 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.710943 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:58.711484 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:58.711686 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:58.714341 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:59807: BlockingRecv error: recv got EOF from 127.9.117.195:59807 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Oct 23 20:12:59 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:59 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:59.466936 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:59.462663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:49267)
1023 20:12:59.462814 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:59.462826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:59.462914 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:59.463059 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:59.463060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:59.463249 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:59.463417 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:59.463419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.464074 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:59.464075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:59.464457 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:59.464458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.464520 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:59.465064 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:59.465068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:59.466860 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
W20251023 20:12:59.467063 19527 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:59.462711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:49267 (local address 127.9.117.254:33491)
1023 20:12:59.462823 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:59.462826 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:59.462894 (+ 68us) server_negotiation.cc:408] Connection header received
1023 20:12:59.462974 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:59.462975 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:59.462988 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:59.463052 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:59.463478 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.464002 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:59.464518 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.464637 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:59.467002 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:49267: BlockingRecv error: recv got EOF from 127.9.117.193:49267 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
Oct 23 20:12:59 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:59 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:59.589280 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:59.584692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:38249)
1023 20:12:59.584906 (+ 214us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:59.584919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:59.584996 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:59.585142 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:59.585143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:59.585330 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:59.585498 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:59.585500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.586165 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:59.586166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:59.586578 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:59.586579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.586656 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:59.587260 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:59.587265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:59.589220 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":111,"threads_started":1}
W20251023 20:12:59.589411 19527 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:59.584779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:38249 (local address 127.9.117.254:33491)
1023 20:12:59.584867 (+ 88us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:59.584870 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:59.584983 (+ 113us) server_negotiation.cc:408] Connection header received
1023 20:12:59.585066 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:59.585066 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:59.585079 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:59.585130 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:59.585576 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.586095 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:59.586659 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.586856 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:59.589337 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:38249: BlockingRecv error: recv got EOF from 127.9.117.194:38249 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Oct 23 20:12:59 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:12:59 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:12:59.719497 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:12:59.714955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:59461)
1023 20:12:59.715113 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1023 20:12:59.715126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:12:59.715197 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:12:59.715387 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:12:59.715388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:12:59.715598 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:12:59.715776 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:59.715779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.716680 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:12:59.716681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:12:59.717115 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:12:59.717117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.717182 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:59.717696 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:12:59.717700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:12:59.719412 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
W20251023 20:12:59.719581 19527 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:12:59.715104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:59461 (local address 127.9.117.254:33491)
1023 20:12:59.715175 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:12:59.715177 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:12:59.715191 (+ 14us) server_negotiation.cc:408] Connection header received
1023 20:12:59.715268 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:12:59.715269 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:12:59.715279 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:12:59.715356 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:12:59.715888 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.716618 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:12:59.717199 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:12:59.717331 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:12:59.719519 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:59461: BlockingRecv error: recv got EOF from 127.9.117.195:59461 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Oct 23 20:13:00 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:00 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:00.478058 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:00.467543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:46399)
1023 20:13:00.467667 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:00.467689 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:00.467773 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:00.468012 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:00.468013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:00.468186 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:00.468351 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:00.468353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.469184 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:00.469186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:00.469594 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:00.469596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.469665 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:00.475769 (+ 6104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:00.475775 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:00.477994 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
W20251023 20:13:00.478312 19531 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:00.467661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:46399 (local address 127.9.117.254:33491)
1023 20:13:00.467868 (+ 207us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:00.467871 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:00.467880 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:13:00.467910 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:00.467911 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:00.467924 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:00.468002 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:00.468456 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.469080 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:00.469685 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.469825 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:00.478232 (+ 8407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:46399: BlockingRecv error: recv got EOF from 127.9.117.193:46399 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":139,"threads_started":1}
Oct 23 20:13:00 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:00 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:00.594527 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:00.589869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:35925)
1023 20:13:00.590084 (+ 215us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:00.590095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:00.590166 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:00.590326 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:00.590326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:00.590545 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:00.590717 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:00.590720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.591401 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:00.591403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:00.591786 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:00.591788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.591868 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:00.592432 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:00.592436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:00.594463 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
W20251023 20:13:00.594679 19531 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:00.589965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:35925 (local address 127.9.117.254:33491)
1023 20:13:00.590016 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:00.590018 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:00.590164 (+ 146us) server_negotiation.cc:408] Connection header received
1023 20:13:00.590231 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:00.590232 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:00.590245 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:00.590314 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:00.590811 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.591318 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:00.591871 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.592073 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:00.594609 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:35925: BlockingRecv error: recv got EOF from 127.9.117.194:35925 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Oct 23 20:13:00 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:00 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:00.724642 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:00.720078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:42059)
1023 20:13:00.720231 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:00.720242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:00.720297 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:00.720446 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:00.720447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:00.720644 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:00.720842 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:00.720845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.721499 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:00.721500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:00.721899 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:00.721900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.721963 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:00.722492 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:00.722497 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:00.724577 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
W20251023 20:13:00.724808 19531 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:00.720205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:42059 (local address 127.9.117.254:33491)
1023 20:13:00.720267 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:00.720269 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:00.720284 (+ 15us) server_negotiation.cc:408] Connection header received
1023 20:13:00.720359 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:00.720360 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:00.720369 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:00.720440 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:00.720928 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.721419 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:00.721977 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:00.722100 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:00.724707 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:42059: BlockingRecv error: recv got EOF from 127.9.117.195:42059 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Oct 23 20:13:01 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:01 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:01.483337 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:01.478693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:52801)
1023 20:13:01.478877 (+ 184us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:01.478888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:01.478946 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:01.479142 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:01.479142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:01.479324 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:01.479491 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:01.479492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.480277 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:01.480279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:01.480757 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:01.480760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.480825 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:01.481380 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:01.481385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:01.483276 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
W20251023 20:13:01.483536 19535 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:01.478875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:52801 (local address 127.9.117.254:33491)
1023 20:13:01.479001 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:01.479003 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:01.479012 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:13:01.479043 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:01.479043 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:01.479053 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:01.479129 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:01.479571 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.480201 (+ 630us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:01.480866 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.481008 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:01.483440 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:52801: BlockingRecv error: recv got EOF from 127.9.117.193:52801 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
Oct 23 20:13:01 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:01 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:01.599660 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:01.595102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:46991)
1023 20:13:01.595228 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:01.595243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:01.595326 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:01.595591 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:01.595592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:01.595780 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:01.595953 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:01.595955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.596850 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:01.596851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:01.597231 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:01.597232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.597302 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:01.597850 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:01.597854 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:01.599579 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
W20251023 20:13:01.599766 19535 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:01.595237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:46991 (local address 127.9.117.254:33491)
1023 20:13:01.595326 (+ 89us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:01.595329 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:01.595340 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:13:01.595396 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:01.595397 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:01.595410 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:01.595560 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:01.596043 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.596773 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:01.597305 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.597519 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:01.599701 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:46991: BlockingRecv error: recv got EOF from 127.9.117.194:46991 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Oct 23 20:13:01 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:01 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:01.729678 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:01.725185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:55831)
1023 20:13:01.725351 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:01.725362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:01.725430 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:01.725626 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:01.725628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:01.725803 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:01.725971 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:01.725973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.726711 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:01.726712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:01.727097 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:01.727099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.727161 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:01.727705 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:01.727709 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:01.729595 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
W20251023 20:13:01.729794 19535 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:01.725302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:55831 (local address 127.9.117.254:33491)
1023 20:13:01.725382 (+ 80us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:01.725384 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:01.725430 (+ 46us) server_negotiation.cc:408] Connection header received
1023 20:13:01.725478 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:01.725478 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:01.725488 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:01.725583 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:01.726078 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.726627 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:01.727172 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:01.727288 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:01.729727 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:55831: BlockingRecv error: recv got EOF from 127.9.117.195:55831 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Oct 23 20:13:02 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:02 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:02.488229 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:02.483920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:51473)
1023 20:13:02.484068 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:02.484079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:02.484139 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:02.484295 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:02.484295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:02.484465 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:02.484638 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:02.484640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.485303 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:02.485303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:02.485685 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:02.485686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.485747 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:02.486322 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:02.486326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:02.488171 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
W20251023 20:13:02.488384 19539 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:02.484015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:51473 (local address 127.9.117.254:33491)
1023 20:13:02.484162 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:02.484165 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:02.484175 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:13:02.484202 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:02.484202 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:02.484213 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:02.484285 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:02.484702 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.485242 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:02.485780 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.485953 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:02.488298 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:51473: BlockingRecv error: recv got EOF from 127.9.117.193:51473 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
Oct 23 20:13:02 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:02 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:02.604913 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:02.600254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:57729)
1023 20:13:02.600415 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:02.600428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:02.600497 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:02.600662 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:02.600663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:02.600874 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:02.601046 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:02.601048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.601720 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:02.601722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:02.602121 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:02.602122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.602186 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:02.602762 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:02.602781 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:02.604800 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
W20251023 20:13:02.605041 19539 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:02.600387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:57729 (local address 127.9.117.254:33491)
1023 20:13:02.600447 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:02.600449 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:02.600478 (+ 29us) server_negotiation.cc:408] Connection header received
1023 20:13:02.600554 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:02.600554 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:02.600565 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:02.600645 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:02.601118 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.601640 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:02.602225 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.602368 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:02.604974 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:57729: BlockingRecv error: recv got EOF from 127.9.117.194:57729 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Oct 23 20:13:02 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:02 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:02.734716 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:02.730309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:47989)
1023 20:13:02.730481 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:02.730494 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:02.730580 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:02.730746 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:02.730747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:02.730987 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:02.731223 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:02.731227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.731879 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:02.731880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:02.732248 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:02.732249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.732308 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:02.732884 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:02.732888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:02.734653 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
W20251023 20:13:02.734843 19539 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:02.730413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:47989 (local address 127.9.117.254:33491)
1023 20:13:02.730476 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:02.730478 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:02.730574 (+ 96us) server_negotiation.cc:408] Connection header received
1023 20:13:02.730642 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:02.730643 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:02.730657 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:02.730724 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:02.731299 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.731788 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:02.732318 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:02.732424 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:02.734760 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:47989: BlockingRecv error: recv got EOF from 127.9.117.195:47989 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Oct 23 20:13:03 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:03 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:03.493492 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:03.488813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:41469)
1023 20:13:03.489009 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:03.489023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:03.489097 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:03.489243 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:03.489244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:03.489423 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:03.489592 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:03.489593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.490329 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:03.490330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:03.490724 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:03.490726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.490790 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:03.491388 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:03.491393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:03.493431 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
W20251023 20:13:03.493649 19543 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:03.488945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:41469 (local address 127.9.117.254:33491)
1023 20:13:03.489086 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:03.489088 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:03.489099 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:13:03.489153 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:03.489153 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:03.489164 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:03.489229 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:03.489676 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.490238 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:03.490799 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.490926 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:03.493561 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:41469: BlockingRecv error: recv got EOF from 127.9.117.193:41469 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
Oct 23 20:13:03 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:03 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:03.610349 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:03.605623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:43011)
1023 20:13:03.605798 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:03.605810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:03.605898 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:03.606030 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:03.606031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:03.606225 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:03.606398 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:03.606401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.607109 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:03.607110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:03.607497 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:03.607499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.607564 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:03.608127 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:03.608131 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:03.610281 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
W20251023 20:13:03.610486 19543 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:03.605753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:43011 (local address 127.9.117.254:33491)
1023 20:13:03.605827 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:03.605830 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:03.605888 (+ 58us) server_negotiation.cc:408] Connection header received
1023 20:13:03.605951 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:03.605952 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:03.605962 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:03.606016 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:03.606487 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.607019 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:03.607606 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.607740 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:03.610404 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:43011: BlockingRecv error: recv got EOF from 127.9.117.194:43011 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 23 20:13:03 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:03 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:03.739295 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:03.735186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:57119)
1023 20:13:03.735382 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:03.735392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:03.735460 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:03.735587 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:03.735587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:03.735765 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:03.735947 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:03.735948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.736578 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:03.736579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:03.736978 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:03.736979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.737036 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:03.737549 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:03.737553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:03.739234 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
W20251023 20:13:03.739439 19543 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:03.735311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:57119 (local address 127.9.117.254:33491)
1023 20:13:03.735387 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:03.735389 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:03.735451 (+ 62us) server_negotiation.cc:408] Connection header received
1023 20:13:03.735512 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:03.735513 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:03.735522 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:03.735571 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:03.736021 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.736506 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:03.737037 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:03.737148 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:03.739352 (+ 2204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:57119: BlockingRecv error: recv got EOF from 127.9.117.195:57119 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Oct 23 20:13:04 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:04 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:04.498361 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:04.494067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:33153)
1023 20:13:04.494219 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:04.494236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:04.494315 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:04.494469 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:04.494470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:04.494648 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:04.494826 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:04.494827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.495459 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:04.495460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:04.495862 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:04.495864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.495924 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:04.496451 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:04.496455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:04.498301 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
W20251023 20:13:04.498512 19547 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:04.494201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:33153 (local address 127.9.117.254:33491)
1023 20:13:04.494346 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:04.494348 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:04.494357 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:13:04.494381 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:04.494382 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:04.494393 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:04.494455 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:04.494896 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.495387 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:04.495925 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.496028 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:04.498427 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:33153: BlockingRecv error: recv got EOF from 127.9.117.193:33153 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
Oct 23 20:13:04 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:04 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:04.615563 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:04.610991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:53345)
1023 20:13:04.611161 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:04.611174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:04.611253 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:04.611395 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:04.611396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:04.611593 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:04.611788 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:04.611790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.612552 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:04.612553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:04.613013 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:04.613016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.613088 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:04.613663 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:04.613667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:04.615472 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
W20251023 20:13:04.615684 19547 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:04.611071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:53345 (local address 127.9.117.254:33491)
1023 20:13:04.611162 (+ 91us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:04.611165 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:04.611243 (+ 78us) server_negotiation.cc:408] Connection header received
1023 20:13:04.611320 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:04.611321 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:04.611334 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:04.611384 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:04.611889 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.612470 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:04.613099 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.613298 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:04.615591 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:53345: BlockingRecv error: recv got EOF from 127.9.117.194:53345 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Oct 23 20:13:04 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:04 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:04.744635 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:04.739885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:33183)
1023 20:13:04.740058 (+ 173us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:04.740075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:04.740168 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:04.740318 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:04.740319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:04.740510 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:04.740693 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:04.740696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.741403 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:04.741405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:04.741793 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:04.741795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.741864 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:04.742467 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:04.742472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:04.744551 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
W20251023 20:13:04.744760 19547 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:04.740038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:33183 (local address 127.9.117.254:33491)
1023 20:13:04.740104 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:04.740106 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:04.740141 (+ 35us) server_negotiation.cc:408] Connection header received
1023 20:13:04.740237 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:04.740238 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:04.740250 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:04.740303 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:04.740808 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.741317 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:04.741894 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:04.742090 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:04.744669 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:33183: BlockingRecv error: recv got EOF from 127.9.117.195:33183 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Oct 23 20:13:05 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:05 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:05.509545 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:05.498841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:42271)
1023 20:13:05.499019 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:05.499030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:05.499094 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:05.499232 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:05.499233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:05.499411 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:05.499580 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:05.499582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.500304 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:05.500305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:05.500708 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:05.500710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.500801 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:05.507555 (+ 6754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:05.507560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:05.509485 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":92,"threads_started":1}
W20251023 20:13:05.509708 19551 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:05.498905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:42271 (local address 127.9.117.254:33491)
1023 20:13:05.499022 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:05.499024 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:05.499093 (+ 69us) server_negotiation.cc:408] Connection header received
1023 20:13:05.499140 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:05.499141 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:05.499150 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:05.499217 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:05.499697 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.500218 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:05.500833 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.500979 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:05.509630 (+ 8651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:42271: BlockingRecv error: recv got EOF from 127.9.117.193:42271 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
Oct 23 20:13:05 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:05 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:05.620663 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:05.616160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:37171)
1023 20:13:05.616330 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:05.616340 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:05.616405 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:05.616571 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:05.616572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:05.616809 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:05.617037 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:05.617040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.617756 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:05.617758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:05.618170 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:05.618172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.618243 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:05.618783 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:05.618787 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:05.620580 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
W20251023 20:13:05.620800 19551 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:05.616238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:37171 (local address 127.9.117.254:33491)
1023 20:13:05.616292 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:05.616294 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:05.616397 (+ 103us) server_negotiation.cc:408] Connection header received
1023 20:13:05.616470 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:05.616471 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:05.616482 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:05.616546 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:05.617130 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.617673 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:05.618255 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.618452 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:05.620701 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:37171: BlockingRecv error: recv got EOF from 127.9.117.194:37171 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Oct 23 20:13:05 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:05 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:05.749933 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:05.745254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:45293)
1023 20:13:05.745459 (+ 205us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:05.745477 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:05.745573 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:05.745749 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:05.745751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:05.745968 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:05.746139 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:05.746142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.746841 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:05.746843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:05.747236 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:05.747238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.747316 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:05.747904 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:05.747909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:05.749847 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
W20251023 20:13:05.750025 19551 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:05.745370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:45293 (local address 127.9.117.254:33491)
1023 20:13:05.745477 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:05.745480 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:05.745589 (+ 109us) server_negotiation.cc:408] Connection header received
1023 20:13:05.745651 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:05.745652 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:05.745666 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:05.745736 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:05.746234 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.746752 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:05.747322 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:05.747509 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:05.749960 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:45293: BlockingRecv error: recv got EOF from 127.9.117.195:45293 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Oct 23 20:13:06 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:06 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:06.514285 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:06.510170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:32793)
1023 20:13:06.510328 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:06.510339 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:06.510399 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:06.510543 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:06.510544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:06.510709 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:06.510888 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:06.510889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.511545 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:06.511546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:06.511943 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:06.511945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.512003 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:06.512514 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:06.512518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:06.514223 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20251023 20:13:06.514405 19555 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:06.510257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:32793 (local address 127.9.117.254:33491)
1023 20:13:06.510405 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:06.510407 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:06.510419 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:13:06.510447 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:06.510448 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:06.510459 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:06.510532 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:06.510970 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.511478 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:06.512014 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.512131 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:06.514330 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:32793: BlockingRecv error: recv got EOF from 127.9.117.193:32793 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
Oct 23 20:13:06 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:06 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:06.625798 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:06.621349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:45245)
1023 20:13:06.621526 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:06.621537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:06.621591 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:06.621761 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:06.621762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:06.621945 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:06.622115 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:06.622118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.622829 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:06.622830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:06.623213 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:06.623215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.623273 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:06.623810 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:06.623814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:06.625733 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
W20251023 20:13:06.625883 19555 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:06.621535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:45245 (local address 127.9.117.254:33491)
1023 20:13:06.621619 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:06.621621 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:06.621654 (+ 33us) server_negotiation.cc:408] Connection header received
1023 20:13:06.621682 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:06.621683 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:06.621693 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:06.621750 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:06.622223 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.622737 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:06.623308 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.623456 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:06.625823 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:45245: BlockingRecv error: recv got EOF from 127.9.117.194:45245 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Oct 23 20:13:06 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:06 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:06.755366 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:06.750455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:43469)
1023 20:13:06.750650 (+ 195us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:06.750660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:06.750731 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:06.750904 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:06.750904 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:06.751064 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:06.751238 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:06.751240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.752294 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:06.752295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:06.752675 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:06.752676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.752763 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:06.753287 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:06.753292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:06.755301 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20251023 20:13:06.755535 19555 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:06.750542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:43469 (local address 127.9.117.254:33491)
1023 20:13:06.750599 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:06.750601 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:06.750736 (+ 135us) server_negotiation.cc:408] Connection header received
1023 20:13:06.750803 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:06.750805 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:06.750818 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:06.750899 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:06.751314 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.752212 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:06.752751 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:06.752983 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:06.755446 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:43469: BlockingRecv error: recv got EOF from 127.9.117.195:43469 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Oct 23 20:13:07 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:07 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:07.519699 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:07.514951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:43091)
1023 20:13:07.515111 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:07.515122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:07.515185 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:07.515449 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:07.515451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:07.515636 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:07.515802 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:07.515804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.516512 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:07.516513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:07.516983 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:07.516985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.517046 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:07.517598 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:07.517602 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:07.519599 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
W20251023 20:13:07.519810 19559 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:07.515096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:43091 (local address 127.9.117.254:33491)
1023 20:13:07.515257 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:07.515259 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:07.515270 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:13:07.515299 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:07.515300 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:07.515311 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:07.515398 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:07.515919 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.516430 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:07.517083 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.517227 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:07.519739 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:43091: BlockingRecv error: recv got EOF from 127.9.117.193:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
Oct 23 20:13:07 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:07 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:07.630615 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:07.626334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:48305)
1023 20:13:07.626508 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:07.626519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:07.626590 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:07.626750 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:07.626750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:07.626922 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:07.627091 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:07.627093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.627729 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:07.627730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:07.628113 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:07.628115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.628175 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:07.628678 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:07.628682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:07.630553 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
W20251023 20:13:07.630740 19559 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:07.626449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:48305 (local address 127.9.117.254:33491)
1023 20:13:07.626522 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:07.626524 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:07.626585 (+ 61us) server_negotiation.cc:408] Connection header received
1023 20:13:07.626667 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:07.626667 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:07.626679 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:07.626729 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:07.627156 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.627653 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:07.628177 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.628284 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:07.630661 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:48305: BlockingRecv error: recv got EOF from 127.9.117.194:48305 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Oct 23 20:13:07 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:07 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.195@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:07.760021 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:07.755878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.195:59789)
1023 20:13:07.756029 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:07.756042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:07.756127 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:07.756277 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:07.756278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:07.756481 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:07.756643 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:07.756645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.757323 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:07.757325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:07.757713 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:07.757714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.757783 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:07.758301 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:07.758305 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:07.759938 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
W20251023 20:13:07.760120 19559 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:07.755983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:59789 (local address 127.9.117.254:33491)
1023 20:13:07.756032 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:07.756034 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:07.756111 (+ 77us) server_negotiation.cc:408] Connection header received
1023 20:13:07.756194 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:07.756195 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:07.756207 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:07.756262 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:07.756756 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.757246 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:07.757801 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:07.758003 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:07.760052 (+ 2049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.195:59789: BlockingRecv error: recv got EOF from 127.9.117.195:59789 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Oct 23 20:13:08 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:08 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.193@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:08.525050 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:08.520432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.193:58011)
1023 20:13:08.520599 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:08.520610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:08.520675 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:08.520844 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:08.520845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:08.521032 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:08.521202 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:08.521204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.522021 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:08.522022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:08.522445 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:08.522447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.522512 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:08.523103 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:08.523108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:08.524990 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
W20251023 20:13:08.525206 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:08.520516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:58011 (local address 127.9.117.254:33491)
1023 20:13:08.520657 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:08.520658 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:08.520669 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:13:08.520744 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:08.520745 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:08.520760 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:08.520839 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:08.521304 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.521939 (+ 635us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:08.522512 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.522644 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:08.525123 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.193:58011: BlockingRecv error: recv got EOF from 127.9.117.193:58011 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
Oct 23 20:13:08 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
Oct 23 20:13:08 dist-test-slave-x62p krb5kdc[18495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.117.194@KRBTEST.COM for oryx/127.9.117.254@KRBTEST.COM, Server not found in Kerberos database
I20251023 20:13:08.635504 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:08.631164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33491 (local address 127.9.117.194:56099)
1023 20:13:08.631297 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:08.631307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:08.631360 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:08.631494 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:08.631494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:08.631666 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:08.631904 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:08.631907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.632617 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:08.632618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:08.633073 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:08.633075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.633145 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:08.633680 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:08.633684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:08.635422 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.117.254:33491: Server oryx/127.9.117.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
W20251023 20:13:08.635622 19563 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:08.631307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:56099 (local address 127.9.117.254:33491)
1023 20:13:08.631368 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:08.631369 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:08.631380 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:13:08.631408 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:08.631409 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:08.631419 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:08.631483 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:08.632005 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.632535 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:08.633158 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.633364 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:08.635534 (+ 2170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.117.194:56099: BlockingRecv error: recv got EOF from 127.9.117.194:56099 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20251023 20:13:08.686038 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 19087
I20251023 20:13:08.691017 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 19239
I20251023 20:13:08.695720 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 19371
I20251023 20:13:08.700374 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 19017
2025-10-23T20:13:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18492 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-10-23T20:13:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:13:08Z Disabled control of system clock
I20251023 20:13:08.728921 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:36223
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251023 20:13:08.810981 19568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:08.811177 19568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:08.811199 19568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:08.812697 19568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:13:08.812768 19568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:08.812781 19568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:08.812808 19568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:13:08.812819 19568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:13:08.814517 19568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:36223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:08.814770 19568 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:08.814988 19568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:08.817816 19576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:08.817816 19574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:08.818032 19568 server_base.cc:1047] running on GCE node
W20251023 20:13:08.817867 19573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:08.818284 19568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:08.818523 19568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:08.819674 19568 hybrid_clock.cc:648] HybridClock initialized: now 1761250388819656 us; error 37 us; skew 500 ppm
I20251023 20:13:08.820875 19568 webserver.cc:492] Webserver started at http://127.9.117.254:35457/ using document root <none> and password file <none>
I20251023 20:13:08.821086 19568 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:08.821146 19568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:08.821274 19568 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:08.822238 19568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "18e326709e824bbea1a7c3052c382617"
format_stamp: "Formatted at 2025-10-23 20:13:08 on dist-test-slave-x62p"
server_key: "e36c4e81c1524f8cdf5a6d49daa057bc"
server_key_iv: "923cb4ed1c477f741690d677cb2b6859"
server_key_version: "encryptionkey@0"
I20251023 20:13:08.822571 19568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "18e326709e824bbea1a7c3052c382617"
format_stamp: "Formatted at 2025-10-23 20:13:08 on dist-test-slave-x62p"
server_key: "e36c4e81c1524f8cdf5a6d49daa057bc"
server_key_iv: "923cb4ed1c477f741690d677cb2b6859"
server_key_version: "encryptionkey@0"
I20251023 20:13:08.823765 19568 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:13:08.824388 19582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:08.824553 19568 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:13:08.824631 19568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "18e326709e824bbea1a7c3052c382617"
format_stamp: "Formatted at 2025-10-23 20:13:08 on dist-test-slave-x62p"
server_key: "e36c4e81c1524f8cdf5a6d49daa057bc"
server_key_iv: "923cb4ed1c477f741690d677cb2b6859"
server_key_version: "encryptionkey@0"
I20251023 20:13:08.824690 19568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:08.834249 19568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:08.834561 19568 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:08.834683 19568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:08.838912 19568 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:36223
I20251023 20:13:08.838951 19634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:36223 every 8 connection(s)
I20251023 20:13:08.839270 19568 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:13:08.839875 19635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:13:08.842367 19635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617: Bootstrap starting.
I20251023 20:13:08.842928 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 19568
I20251023 20:13:08.842949 19635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617: Neither blocks nor log segments found. Creating new log.
I20251023 20:13:08.843019 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:13:08.843190 19635 log.cc:826] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617: Log is configured to *not* fsync() on all Append() calls
I20251023 20:13:08.843204 9687 external_mini_cluster.cc:1468] Setting key c94664abeb7865a6f5704763f08a7d96
I20251023 20:13:08.843933 19635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617: No bootstrap required, opened a new log
I20251023 20:13:08.845608 19635 raft_consensus.cc:359] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e326709e824bbea1a7c3052c382617" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36223 } }
I20251023 20:13:08.845753 19635 raft_consensus.cc:385] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:13:08.845779 19635 raft_consensus.cc:740] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e326709e824bbea1a7c3052c382617, State: Initialized, Role: FOLLOWER
I20251023 20:13:08.845875 19635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e326709e824bbea1a7c3052c382617" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36223 } }
I20251023 20:13:08.845932 19635 raft_consensus.cc:399] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:13:08.845956 19635 raft_consensus.cc:493] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:13:08.846009 19635 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:13:08.846612 19635 raft_consensus.cc:515] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e326709e824bbea1a7c3052c382617" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36223 } }
I20251023 20:13:08.846725 19635 leader_election.cc:304] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18e326709e824bbea1a7c3052c382617; no voters:
I20251023 20:13:08.846871 19635 leader_election.cc:290] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:13:08.847043 19641 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:13:08.847107 19635 sys_catalog.cc:565] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:13:08.847456 19641 raft_consensus.cc:697] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [term 1 LEADER]: Becoming Leader. State: Replica: 18e326709e824bbea1a7c3052c382617, State: Running, Role: LEADER
I20251023 20:13:08.847651 19641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e326709e824bbea1a7c3052c382617" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36223 } }
I20251023 20:13:08.848074 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:08.843713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.9.117.254:36223)
1023 20:13:08.843970 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:08.843978 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:08.844002 (+ 24us) server_negotiation.cc:408] Connection header received
1023 20:13:08.844386 (+ 384us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:08.844395 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:08.844469 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:08.844628 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:08.844995 (+ 367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.846110 (+ 1115us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:08.846698 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:08.846971 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:08.847384 (+ 413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:08.847390 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:08.847391 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:13:08.847606 (+ 215us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:08.847647 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:13:08.847654 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:08.847731 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:08.847833 (+ 102us) server_negotiation.cc:300] Negotiation successful
1023 20:13:08.847888 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20251023 20:13:08.848251 19640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18e326709e824bbea1a7c3052c382617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e326709e824bbea1a7c3052c382617" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36223 } } }
I20251023 20:13:08.848328 19640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [sys.catalog]: This master's current role is: LEADER
I20251023 20:13:08.848832 19649 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:13:08.848526 19642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e326709e824bbea1a7c3052c382617. Latest consensus state: current_term: 1 leader_uuid: "18e326709e824bbea1a7c3052c382617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e326709e824bbea1a7c3052c382617" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36223 } } }
I20251023 20:13:08.849113 19642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617 [sys.catalog]: This master's current role is: LEADER
I20251023 20:13:08.849545 19649 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:13:08.850884 19649 catalog_manager.cc:1357] Generated new cluster ID: 05e1eb9af15a452fae9d6411d25829cc
I20251023 20:13:08.850929 19649 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:13:08.857062 19649 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:13:08.857615 19649 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:13:08.869148 19649 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 18e326709e824bbea1a7c3052c382617: Generated new TSK 0
I20251023 20:13:08.869349 19649 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:13:08.876523 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:36223
--builtin_ntp_servers=127.9.117.212:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251023 20:13:08.961390 19659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:08.961583 19659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:08.961628 19659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:08.963202 19659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:08.963250 19659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:08.963290 19659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:13:08.964991 19659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:08.965255 19659 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:08.965507 19659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:08.968425 19665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:08.968418 19667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:08.968421 19664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:08.968703 19659 server_base.cc:1047] running on GCE node
I20251023 20:13:08.968904 19659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:08.969125 19659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:08.970274 19659 hybrid_clock.cc:648] HybridClock initialized: now 1761250388970255 us; error 35 us; skew 500 ppm
I20251023 20:13:08.971557 19659 webserver.cc:492] Webserver started at http://127.9.117.193:37049/ using document root <none> and password file <none>
I20251023 20:13:08.971796 19659 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:08.971843 19659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:08.971963 19659 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:08.973017 19659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "5d65f6e230d04c20b6874da40d7ccbbd"
format_stamp: "Formatted at 2025-10-23 20:13:08 on dist-test-slave-x62p"
server_key: "526d5548a573b68072de4356d3981663"
server_key_iv: "3ec2caa1b65c6a2f2f039f1517e769e1"
server_key_version: "encryptionkey@0"
I20251023 20:13:08.973335 19659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "5d65f6e230d04c20b6874da40d7ccbbd"
format_stamp: "Formatted at 2025-10-23 20:13:08 on dist-test-slave-x62p"
server_key: "526d5548a573b68072de4356d3981663"
server_key_iv: "3ec2caa1b65c6a2f2f039f1517e769e1"
server_key_version: "encryptionkey@0"
I20251023 20:13:08.974537 19659 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:13:08.975327 19673 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:08.975517 19659 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:13:08.975591 19659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "5d65f6e230d04c20b6874da40d7ccbbd"
format_stamp: "Formatted at 2025-10-23 20:13:08 on dist-test-slave-x62p"
server_key: "526d5548a573b68072de4356d3981663"
server_key_iv: "3ec2caa1b65c6a2f2f039f1517e769e1"
server_key_version: "encryptionkey@0"
I20251023 20:13:08.975689 19659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:08.990970 19659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:08.991253 19659 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:08.991385 19659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:08.991650 19659 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:08.992046 19659 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:08.992090 19659 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:08.992129 19659 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:08.992161 19659 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.000229 19659 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:39401
I20251023 20:13:09.000298 19786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:39401 every 8 connection(s)
I20251023 20:13:09.000645 19659 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:13:09.001682 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 19659
I20251023 20:13:09.001812 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:13:09.002025 9687 external_mini_cluster.cc:1468] Setting key 78477f628f599caa58f4697cf9b23c49
I20251023 20:13:09.003170 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:36223
--builtin_ntp_servers=127.9.117.212:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251023 20:13:09.005549 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.001462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:45345 (local address 127.9.117.254:36223)
1023 20:13:09.001549 (+ 87us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.001552 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.001777 (+ 225us) server_negotiation.cc:408] Connection header received
1023 20:13:09.002602 (+ 825us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.002604 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.002616 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.002669 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.003221 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.003742 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:09.004374 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.004550 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:09.005204 (+ 654us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:09.005207 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:09.005207 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:13:09.005212 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:09.005226 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:13:09.005230 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:09.005281 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:09.005409 (+ 128us) server_negotiation.cc:300] Negotiation successful
1023 20:13:09.005447 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251023 20:13:09.006173 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.001416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:36223 (local address 127.9.117.193:45345)
1023 20:13:09.001668 (+ 252us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:09.001698 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:09.002472 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:09.002734 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:09.002738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:09.002750 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:09.003130 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:09.003134 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.003853 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:09.003854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:09.004295 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:09.004297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.004430 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:09.005086 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:13:09.005087 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:09.005107 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:13:09.005126 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:09.005309 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:09.005314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:09.005317 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:13:09.005871 (+ 554us) client_negotiation.cc:240] Negotiation successful
1023 20:13:09.005978 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":104,"threads_started":1}
I20251023 20:13:09.006893 19787 heartbeater.cc:344] Connected to a master server at 127.9.117.254:36223
I20251023 20:13:09.006987 19787 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:09.007194 19787 heartbeater.cc:507] Master 127.9.117.254:36223 requested a full tablet report, sending...
I20251023 20:13:09.007661 19599 ts_manager.cc:194] Registered new tserver with Master: 5d65f6e230d04c20b6874da40d7ccbbd (127.9.117.193:39401)
I20251023 20:13:09.008316 19599 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.193:45345
W20251023 20:13:09.085778 19790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:09.085970 19790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:09.086007 19790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:09.087431 19790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:09.087473 19790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:09.087513 19790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:13:09.089072 19790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:09.089318 19790 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:09.089552 19790 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:09.092193 19796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:09.092219 19795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.092250 19790 server_base.cc:1047] running on GCE node
W20251023 20:13:09.092219 19798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.092599 19790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:09.092839 19790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:09.093978 19790 hybrid_clock.cc:648] HybridClock initialized: now 1761250389093963 us; error 36 us; skew 500 ppm
I20251023 20:13:09.095140 19790 webserver.cc:492] Webserver started at http://127.9.117.194:45115/ using document root <none> and password file <none>
I20251023 20:13:09.095346 19790 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:09.095395 19790 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:09.095515 19790 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:09.096489 19790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "ba1dc0998eb64fe0bab72fec9126a0ea"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "ab351841c5ec8a4f5badd530c6ff1a95"
server_key_iv: "e36c7c9c61879810ebae7ed4fbabb8c5"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.096853 19790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "ba1dc0998eb64fe0bab72fec9126a0ea"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "ab351841c5ec8a4f5badd530c6ff1a95"
server_key_iv: "e36c7c9c61879810ebae7ed4fbabb8c5"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.098042 19790 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:13:09.098748 19804 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.098913 19790 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.098984 19790 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "ba1dc0998eb64fe0bab72fec9126a0ea"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "ab351841c5ec8a4f5badd530c6ff1a95"
server_key_iv: "e36c7c9c61879810ebae7ed4fbabb8c5"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.099045 19790 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:09.108453 19790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:09.108711 19790 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:09.108899 19790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:09.109133 19790 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:09.109458 19790 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:09.109491 19790 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.109526 19790 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:09.109555 19790 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.115465 19790 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:38617
I20251023 20:13:09.115525 19917 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:38617 every 8 connection(s)
I20251023 20:13:09.115828 19790 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:13:09.118474 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 19790
I20251023 20:13:09.118580 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:13:09.118808 9687 external_mini_cluster.cc:1468] Setting key 811f326befc6a0657187ff1aecd530bf
I20251023 20:13:09.119921 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:36223
--builtin_ntp_servers=127.9.117.212:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251023 20:13:09.120354 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.116611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:45985 (local address 127.9.117.254:36223)
1023 20:13:09.116684 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.116686 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.116906 (+ 220us) server_negotiation.cc:408] Connection header received
1023 20:13:09.117391 (+ 485us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.117393 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.117403 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.117469 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.117938 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.118589 (+ 651us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:09.119316 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.119496 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:09.120092 (+ 596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:09.120096 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:09.120097 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:13:09.120102 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:09.120117 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:13:09.120121 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:09.120149 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:09.120264 (+ 115us) server_negotiation.cc:300] Negotiation successful
1023 20:13:09.120293 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251023 20:13:09.120563 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.116587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:36223 (local address 127.9.117.194:45985)
1023 20:13:09.116809 (+ 222us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:09.116834 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:09.117314 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:09.117524 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:09.117528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:09.117536 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:09.117842 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:09.117844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.118714 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:09.118716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:09.119239 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:09.119242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.119392 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:09.119931 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:13:09.119932 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:09.119962 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:13:09.119986 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:09.120189 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:09.120193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:09.120195 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:13:09.120262 (+ 67us) client_negotiation.cc:240] Negotiation successful
1023 20:13:09.120354 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20251023 20:13:09.121129 19918 heartbeater.cc:344] Connected to a master server at 127.9.117.254:36223
I20251023 20:13:09.121225 19918 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:09.121474 19918 heartbeater.cc:507] Master 127.9.117.254:36223 requested a full tablet report, sending...
I20251023 20:13:09.121927 19599 ts_manager.cc:194] Registered new tserver with Master: ba1dc0998eb64fe0bab72fec9126a0ea (127.9.117.194:38617)
I20251023 20:13:09.122303 19599 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.194:45985
W20251023 20:13:09.203576 19921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:09.203755 19921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:09.203778 19921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:09.205333 19921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:09.205377 19921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:09.205404 19921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:13:09.206954 19921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:09.207183 19921 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:09.207419 19921 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:09.210052 19926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:09.210067 19927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:09.210052 19929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.210283 19921 server_base.cc:1047] running on GCE node
I20251023 20:13:09.210441 19921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:09.210672 19921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:09.211827 19921 hybrid_clock.cc:648] HybridClock initialized: now 1761250389211812 us; error 32 us; skew 500 ppm
I20251023 20:13:09.213019 19921 webserver.cc:492] Webserver started at http://127.9.117.195:33711/ using document root <none> and password file <none>
I20251023 20:13:09.213215 19921 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:09.213255 19921 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:09.213372 19921 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:09.214349 19921 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "b59ddbc61df04ac98b38917af85f33df"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "c590c24d7db687a24698033119053a2a"
server_key_iv: "ecbe3f35b0022d8876362f1e6c6df3cf"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.214658 19921 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "b59ddbc61df04ac98b38917af85f33df"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "c590c24d7db687a24698033119053a2a"
server_key_iv: "ecbe3f35b0022d8876362f1e6c6df3cf"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.215852 19921 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:13:09.216571 19935 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.216773 19921 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:13:09.216854 19921 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "b59ddbc61df04ac98b38917af85f33df"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "c590c24d7db687a24698033119053a2a"
server_key_iv: "ecbe3f35b0022d8876362f1e6c6df3cf"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.216925 19921 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:09.230901 19921 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:09.231199 19921 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:09.231316 19921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:09.231557 19921 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:09.231913 19921 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:09.231952 19921 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.231987 19921 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:09.232018 19921 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.238215 19921 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:33797
I20251023 20:13:09.238279 20048 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:33797 every 8 connection(s)
I20251023 20:13:09.238616 19921 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:13:09.243088 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.239407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:37983 (local address 127.9.117.254:36223)
1023 20:13:09.239471 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.239475 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.239694 (+ 219us) server_negotiation.cc:408] Connection header received
1023 20:13:09.240265 (+ 571us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.240266 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.240280 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.240344 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.240892 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.241385 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:09.241977 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.242143 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:09.242791 (+ 648us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:09.242798 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:09.242798 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:13:09.242804 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:09.242818 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:13:09.242822 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:09.242872 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:09.242969 (+ 97us) server_negotiation.cc:300] Negotiation successful
1023 20:13:09.243002 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251023 20:13:09.243238 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.239396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:36223 (local address 127.9.117.195:37983)
1023 20:13:09.239581 (+ 185us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:09.239606 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:09.240157 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:09.240425 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:09.240429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:09.240438 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:09.240790 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:09.240794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.241481 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:09.241482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:09.241883 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:09.241884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:09.242042 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:09.242661 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:13:09.242662 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:09.242688 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:13:09.242707 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:09.242877 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:09.242882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:09.242885 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:13:09.242969 (+ 84us) client_negotiation.cc:240] Negotiation successful
1023 20:13:09.243056 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20251023 20:13:09.243731 20049 heartbeater.cc:344] Connected to a master server at 127.9.117.254:36223
I20251023 20:13:09.243844 20049 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:09.244053 20049 heartbeater.cc:507] Master 127.9.117.254:36223 requested a full tablet report, sending...
I20251023 20:13:09.244444 19599 ts_manager.cc:194] Registered new tserver with Master: b59ddbc61df04ac98b38917af85f33df (127.9.117.195:33797)
I20251023 20:13:09.244926 19599 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.195:37983
I20251023 20:13:09.245469 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 19921
I20251023 20:13:09.245570 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:13:09.245823 9687 external_mini_cluster.cc:1468] Setting key efbae867579cad886cb2291b332f1000
I20251023 20:13:09.246644 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251023 20:13:09.248493 19639 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:09.247961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.9.117.254:36223)
1023 20:13:09.248024 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.248026 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.248135 (+ 109us) server_negotiation.cc:408] Connection header received
1023 20:13:09.248207 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.248208 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.248219 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.248271 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.248423 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39022: BlockingRecv error: recv got EOF from 127.0.0.1:39022 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20251023 20:13:09.248533 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 19659
I20251023 20:13:09.254376 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 19790
I20251023 20:13:09.259331 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 19921
I20251023 20:13:09.263785 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 19568
2025-10-23T20:13:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (561 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-10-23T20:13:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:13:09Z Disabled control of system clock
I20251023 20:13:09.290566 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40229
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251023 20:13:09.372975 20060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:09.373188 20060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:09.373212 20060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:09.374727 20060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:13:09.374774 20060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:09.374787 20060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:09.374800 20060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:13:09.374814 20060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:13:09.376368 20060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:40229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:09.376621 20060 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:09.376878 20060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:13:09.379822 20060 server_base.cc:1047] running on GCE node
W20251023 20:13:09.379739 20065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:09.379740 20066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:09.379755 20068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.381155 20060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:09.381441 20060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:09.382583 20060 hybrid_clock.cc:648] HybridClock initialized: now 1761250389382566 us; error 36 us; skew 500 ppm
I20251023 20:13:09.383867 20060 webserver.cc:492] Webserver started at http://127.9.117.254:45799/ using document root <none> and password file <none>
I20251023 20:13:09.384088 20060 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:09.384136 20060 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:09.384258 20060 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:09.385346 20060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "31d882f2980144beb9707827f9b1f35c"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "792961c78a77d07d7f1b84353e102b3c"
server_key_iv: "366f7fb2f385a80ffa90d2ac02772065"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.385679 20060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "31d882f2980144beb9707827f9b1f35c"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "792961c78a77d07d7f1b84353e102b3c"
server_key_iv: "366f7fb2f385a80ffa90d2ac02772065"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.386972 20060 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:13:09.387670 20074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.387877 20060 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:13:09.387959 20060 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "31d882f2980144beb9707827f9b1f35c"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "792961c78a77d07d7f1b84353e102b3c"
server_key_iv: "366f7fb2f385a80ffa90d2ac02772065"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.388025 20060 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:09.391737 20060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:09.392053 20060 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:09.392174 20060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:09.396628 20060 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:40229
I20251023 20:13:09.396677 20126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:40229 every 8 connection(s)
I20251023 20:13:09.397059 20060 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:13:09.397686 20127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:13:09.400269 20127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c: Bootstrap starting.
I20251023 20:13:09.400930 20127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c: Neither blocks nor log segments found. Creating new log.
I20251023 20:13:09.401176 20127 log.cc:826] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c: Log is configured to *not* fsync() on all Append() calls
I20251023 20:13:09.401914 20127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c: No bootstrap required, opened a new log
I20251023 20:13:09.403205 20127 raft_consensus.cc:359] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d882f2980144beb9707827f9b1f35c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40229 } }
I20251023 20:13:09.403343 20127 raft_consensus.cc:385] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:13:09.403391 20127 raft_consensus.cc:740] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d882f2980144beb9707827f9b1f35c, State: Initialized, Role: FOLLOWER
I20251023 20:13:09.403524 20127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d882f2980144beb9707827f9b1f35c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40229 } }
I20251023 20:13:09.403637 20127 raft_consensus.cc:399] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:13:09.403681 20127 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:13:09.403734 20127 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:13:09.404345 20127 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d882f2980144beb9707827f9b1f35c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40229 } }
I20251023 20:13:09.404458 20127 leader_election.cc:304] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d882f2980144beb9707827f9b1f35c; no voters:
I20251023 20:13:09.404438 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 20060
I20251023 20:13:09.404563 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:13:09.404635 20127 leader_election.cc:290] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:13:09.404708 20130 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:13:09.404831 9687 external_mini_cluster.cc:1468] Setting key 53034beda05dfa575531ae1f143a0116
I20251023 20:13:09.404937 20127 sys_catalog.cc:565] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:13:09.404947 20130 raft_consensus.cc:697] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [term 1 LEADER]: Becoming Leader. State: Replica: 31d882f2980144beb9707827f9b1f35c, State: Running, Role: LEADER
I20251023 20:13:09.405083 20130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d882f2980144beb9707827f9b1f35c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40229 } }
I20251023 20:13:09.405579 20131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d882f2980144beb9707827f9b1f35c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d882f2980144beb9707827f9b1f35c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40229 } } }
I20251023 20:13:09.405601 20132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d882f2980144beb9707827f9b1f35c. Latest consensus state: current_term: 1 leader_uuid: "31d882f2980144beb9707827f9b1f35c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d882f2980144beb9707827f9b1f35c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40229 } } }
I20251023 20:13:09.405745 20131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [sys.catalog]: This master's current role is: LEADER
I20251023 20:13:09.405764 20132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c [sys.catalog]: This master's current role is: LEADER
I20251023 20:13:09.407186 20137 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:13:09.407543 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.405366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.9.117.254:40229)
1023 20:13:09.405642 (+ 276us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.405651 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.405675 (+ 24us) server_negotiation.cc:408] Connection header received
1023 20:13:09.405798 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.405804 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.405845 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.406243 (+ 398us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.406961 (+ 718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:09.406964 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:09.406964 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:13:09.406967 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:09.406987 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:13:09.406992 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:09.407076 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:09.407289 (+ 213us) server_negotiation.cc:300] Negotiation successful
1023 20:13:09.407352 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20251023 20:13:09.408012 20137 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:13:09.409474 20137 catalog_manager.cc:1357] Generated new cluster ID: 1fa74591cf6b432b89e3a6115df030ab
I20251023 20:13:09.409583 20137 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:13:09.414783 20137 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:13:09.414939 20137 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:13:09.426491 20137 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 31d882f2980144beb9707827f9b1f35c: Generated new TSK 0
I20251023 20:13:09.426726 20137 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:13:09.428395 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40229
--builtin_ntp_servers=127.9.117.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251023 20:13:09.510164 20151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:09.510360 20151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:09.510387 20151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:09.511932 20151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:09.511987 20151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:09.512017 20151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:13:09.513558 20151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:09.513813 20151 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:09.514068 20151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:09.516709 20159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:09.516939 20157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.516892 20151 server_base.cc:1047] running on GCE node
W20251023 20:13:09.516816 20156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.517261 20151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:09.517490 20151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:09.518630 20151 hybrid_clock.cc:648] HybridClock initialized: now 1761250389518605 us; error 38 us; skew 500 ppm
I20251023 20:13:09.520004 20151 webserver.cc:492] Webserver started at http://127.9.117.193:39441/ using document root <none> and password file <none>
I20251023 20:13:09.520215 20151 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:09.520262 20151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:09.520391 20151 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:09.521425 20151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "3e58fc3ed0554e969eaff96465dbf49e"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "64b32966ddc75813397d048984ccfb92"
server_key_iv: "4b76520d9ada982bc3cc9563b1e08dcd"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.521756 20151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "3e58fc3ed0554e969eaff96465dbf49e"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "64b32966ddc75813397d048984ccfb92"
server_key_iv: "4b76520d9ada982bc3cc9563b1e08dcd"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.523094 20151 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:13:09.523792 20165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.524027 20151 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:13:09.524127 20151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "3e58fc3ed0554e969eaff96465dbf49e"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "64b32966ddc75813397d048984ccfb92"
server_key_iv: "4b76520d9ada982bc3cc9563b1e08dcd"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.524199 20151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:09.529551 20151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:09.529903 20151 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:09.530030 20151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:09.530263 20151 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:09.530634 20151 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:09.530686 20151 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.530730 20151 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:09.530748 20151 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.538507 20151 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:42629
I20251023 20:13:09.538571 20278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:42629 every 8 connection(s)
I20251023 20:13:09.538941 20151 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:13:09.541970 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.539769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:35675 (local address 127.9.117.254:40229)
1023 20:13:09.539824 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.539826 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.540078 (+ 252us) server_negotiation.cc:408] Connection header received
1023 20:13:09.540560 (+ 482us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.540561 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.540567 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.540636 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.541680 (+ 1044us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:09.541682 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:09.541682 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:13:09.541688 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:09.541707 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:13:09.541711 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:09.541784 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:09.541869 (+ 85us) server_negotiation.cc:300] Negotiation successful
1023 20:13:09.541889 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251023 20:13:09.542075 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.539760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40229 (local address 127.9.117.193:35675)
1023 20:13:09.539962 (+ 202us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:09.539987 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:09.540463 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:09.540694 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:09.540697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:09.540712 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:09.541515 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:13:09.541516 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:09.541539 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:13:09.541559 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:09.541782 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:09.541784 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:09.541786 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:13:09.541863 (+ 77us) client_negotiation.cc:240] Negotiation successful
1023 20:13:09.541905 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20251023 20:13:09.542580 20279 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40229
I20251023 20:13:09.542706 20279 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:09.542873 20279 heartbeater.cc:507] Master 127.9.117.254:40229 requested a full tablet report, sending...
I20251023 20:13:09.543035 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 20151
I20251023 20:13:09.543140 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:13:09.543320 9687 external_mini_cluster.cc:1468] Setting key 4e99034cf7ed723913572ea3aee6d1b8
I20251023 20:13:09.543365 20091 ts_manager.cc:194] Registered new tserver with Master: 3e58fc3ed0554e969eaff96465dbf49e (127.9.117.193:42629)
I20251023 20:13:09.544294 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40229
--builtin_ntp_servers=127.9.117.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251023 20:13:09.630157 20282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:09.630332 20282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:09.630354 20282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:09.631791 20282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:09.631834 20282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:09.631858 20282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:13:09.633303 20282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:09.633549 20282 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:09.633795 20282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:09.636469 20287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:09.636601 20290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.636543 20282 server_base.cc:1047] running on GCE node
W20251023 20:13:09.636471 20288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.636869 20282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:09.637087 20282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:09.638231 20282 hybrid_clock.cc:648] HybridClock initialized: now 1761250389638210 us; error 37 us; skew 500 ppm
I20251023 20:13:09.639412 20282 webserver.cc:492] Webserver started at http://127.9.117.194:35579/ using document root <none> and password file <none>
I20251023 20:13:09.639622 20282 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:09.639671 20282 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:09.639933 20282 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:09.640973 20282 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "e9d700c1edb24c559665058febce590c"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "74328c8eb7cd16c79f7c916830c782f0"
server_key_iv: "e0ba85b553d205889de1549814986de4"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.641320 20282 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "e9d700c1edb24c559665058febce590c"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "74328c8eb7cd16c79f7c916830c782f0"
server_key_iv: "e0ba85b553d205889de1549814986de4"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.642573 20282 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:13:09.643286 20296 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.643445 20282 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:13:09.643515 20282 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "e9d700c1edb24c559665058febce590c"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "74328c8eb7cd16c79f7c916830c782f0"
server_key_iv: "e0ba85b553d205889de1549814986de4"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.643579 20282 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:09.647118 20282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:09.647372 20282 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:09.647493 20282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:09.647689 20282 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:09.647970 20282 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:09.648005 20282 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.648036 20282 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:09.648069 20282 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.654209 20282 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:44201
I20251023 20:13:09.654263 20409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:44201 every 8 connection(s)
I20251023 20:13:09.654569 20282 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:13:09.657243 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.655273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:35577 (local address 127.9.117.254:40229)
1023 20:13:09.655357 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.655360 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.655535 (+ 175us) server_negotiation.cc:408] Connection header received
1023 20:13:09.656087 (+ 552us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.656089 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.656099 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.656154 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.656972 (+ 818us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:09.656974 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:09.656974 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:13:09.656979 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:09.656998 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:13:09.657002 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:09.657056 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:09.657139 (+ 83us) server_negotiation.cc:300] Negotiation successful
1023 20:13:09.657161 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:13:09.657435 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.655228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40229 (local address 127.9.117.194:35577)
1023 20:13:09.655429 (+ 201us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:09.655455 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:09.655990 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:09.656219 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:09.656224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:09.656233 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:09.656828 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:13:09.656829 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:09.656856 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:13:09.656879 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:09.657060 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:09.657065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:09.657068 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:13:09.657171 (+ 103us) client_negotiation.cc:240] Negotiation successful
1023 20:13:09.657227 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20251023 20:13:09.657936 20410 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40229
I20251023 20:13:09.658022 20410 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:09.658144 20410 heartbeater.cc:507] Master 127.9.117.254:40229 requested a full tablet report, sending...
I20251023 20:13:09.658490 20091 ts_manager.cc:194] Registered new tserver with Master: e9d700c1edb24c559665058febce590c (127.9.117.194:44201)
I20251023 20:13:09.659260 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 20282
I20251023 20:13:09.659359 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:13:09.659536 9687 external_mini_cluster.cc:1468] Setting key 5e18a6a49de73cedb556bb421aeda8da
I20251023 20:13:09.660450 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40229
--builtin_ntp_servers=127.9.117.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251023 20:13:09.741071 20413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:09.741269 20413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:09.741293 20413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:09.742762 20413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:09.742811 20413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:09.742841 20413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:13:09.744369 20413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:09.744634 20413 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:09.744899 20413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:09.748148 20418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.748167 20413 server_base.cc:1047] running on GCE node
W20251023 20:13:09.748193 20419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:09.748672 20421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:09.748930 20413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:09.749145 20413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:09.750293 20413 hybrid_clock.cc:648] HybridClock initialized: now 1761250389750264 us; error 44 us; skew 500 ppm
I20251023 20:13:09.751636 20413 webserver.cc:492] Webserver started at http://127.9.117.195:46009/ using document root <none> and password file <none>
I20251023 20:13:09.751856 20413 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:09.751900 20413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:09.752038 20413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:09.753132 20413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "b8cdaf40075f4c4cb964c98a25abc6e0"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "14985532c09ef5f41e2d0810c30abcc6"
server_key_iv: "f6ee056932d3fca83861d561b470bb46"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.753460 20413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "b8cdaf40075f4c4cb964c98a25abc6e0"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "14985532c09ef5f41e2d0810c30abcc6"
server_key_iv: "f6ee056932d3fca83861d561b470bb46"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.754841 20413 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:13:09.755653 20427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.755888 20413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:13:09.755972 20413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "b8cdaf40075f4c4cb964c98a25abc6e0"
format_stamp: "Formatted at 2025-10-23 20:13:09 on dist-test-slave-x62p"
server_key: "14985532c09ef5f41e2d0810c30abcc6"
server_key_iv: "f6ee056932d3fca83861d561b470bb46"
server_key_version: "encryptionkey@0"
I20251023 20:13:09.756044 20413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:09.759907 20413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:09.760195 20413 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:09.760314 20413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:09.760512 20413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:09.760875 20413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:09.760912 20413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.760949 20413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:09.760979 20413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:09.767206 20413 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:43435
I20251023 20:13:09.767264 20540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:43435 every 8 connection(s)
I20251023 20:13:09.767575 20413 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:13:09.770154 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.768308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:38461 (local address 127.9.117.254:40229)
1023 20:13:09.768377 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.768379 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.768575 (+ 196us) server_negotiation.cc:408] Connection header received
1023 20:13:09.769151 (+ 576us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.769153 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.769158 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.769220 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.769943 (+ 723us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:09.769944 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:09.769944 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:13:09.769948 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:09.769965 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:13:09.769968 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:09.770015 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:09.770081 (+ 66us) server_negotiation.cc:300] Negotiation successful
1023 20:13:09.770093 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251023 20:13:09.770326 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:09.768276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:40229 (local address 127.9.117.195:38461)
1023 20:13:09.768445 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:09.768472 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:09.769058 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:09.769277 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:09.769281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:09.769290 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:09.769824 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:13:09.769824 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:09.769844 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:13:09.769862 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:09.770023 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:09.770026 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:09.770028 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:13:09.770096 (+ 68us) client_negotiation.cc:240] Negotiation successful
1023 20:13:09.770137 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20251023 20:13:09.770853 20541 heartbeater.cc:344] Connected to a master server at 127.9.117.254:40229
I20251023 20:13:09.770947 20541 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:09.771090 20541 heartbeater.cc:507] Master 127.9.117.254:40229 requested a full tablet report, sending...
I20251023 20:13:09.771437 20091 ts_manager.cc:194] Registered new tserver with Master: b8cdaf40075f4c4cb964c98a25abc6e0 (127.9.117.195:43435)
I20251023 20:13:09.775043 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 20413
I20251023 20:13:09.775147 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:13:09.775324 9687 external_mini_cluster.cc:1468] Setting key 3eb27f18eab4dfde3407223ae92096ec
I20251023 20:13:09.776257 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251023 20:13:09.778081 20134 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:13:09.777666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.9.117.254:40229)
1023 20:13:09.777718 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:09.777720 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:09.777811 (+ 91us) server_negotiation.cc:408] Connection header received
1023 20:13:09.777857 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:09.777858 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:09.777864 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:09.777919 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:09.778023 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50842: BlockingRecv error: recv got EOF from 127.0.0.1:50842 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:13:09.778184 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 20151
I20251023 20:13:09.783746 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 20282
I20251023 20:13:09.788645 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 20413
I20251023 20:13:09.793174 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 20060
2025-10-23T20:13:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:13:09 dist-test-slave-x62p krb5kdc[20551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:13:09 dist-test-slave-x62p krb5kdc[20551](info): set up 2 sockets
Oct 23 20:13:09 dist-test-slave-x62p krb5kdc[20551](info): commencing operation
krb5kdc: starting...
W20251023 20:13:11.793642 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.981s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:13:11 dist-test-slave-x62p krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:13:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:13:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:13:11.948333 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37375
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:13:12.029866 20567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:12.030090 20567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:12.030129 20567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:12.031643 20567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:13:12.031701 20567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:12.031725 20567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:12.031745 20567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:13:12.031766 20567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:13:12.033565 20567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:12.033839 20567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:12.034039 20567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:12.036633 20573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:12.036648 20572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:12.036814 20567 server_base.cc:1047] running on GCE node
W20251023 20:13:12.036661 20575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:12.037173 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:12.037431 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:12.038568 20567 hybrid_clock.cc:648] HybridClock initialized: now 1761250392038551 us; error 46 us; skew 500 ppm
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:13:12.041081 20567 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:13:12.041785 20567 webserver.cc:492] Webserver started at http://127.9.117.254:33663/ using document root <none> and password file <none>
I20251023 20:13:12.041986 20567 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:12.042022 20567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:12.042121 20567 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:12.043082 20567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "803da47cc420de11e94509b3b3469600"
server_key_iv: "91a46dd25e87b118815d3ab2314d15e8"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.043354 20567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "803da47cc420de11e94509b3b3469600"
server_key_iv: "91a46dd25e87b118815d3ab2314d15e8"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.044530 20567 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251023 20:13:12.045274 20582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.045455 20567 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.045523 20567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "803da47cc420de11e94509b3b3469600"
server_key_iv: "91a46dd25e87b118815d3ab2314d15e8"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.045576 20567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:12.062287 20567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:12.064201 20567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:12.064319 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:12.068428 20567 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37375
I20251023 20:13:12.068483 20634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37375 every 8 connection(s)
I20251023 20:13:12.068845 20567 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:13:12.069494 20635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:13:12.072155 20635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b: Bootstrap starting.
I20251023 20:13:12.072790 20635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b: Neither blocks nor log segments found. Creating new log.
I20251023 20:13:12.073104 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 20567
I20251023 20:13:12.073109 20635 log.cc:826] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b: Log is configured to *not* fsync() on all Append() calls
I20251023 20:13:12.073216 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:13:12.073414 9687 external_mini_cluster.cc:1468] Setting key aa178e56ee0af43bc36f2399996cbc2a
I20251023 20:13:12.074005 20635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b: No bootstrap required, opened a new log
I20251023 20:13:12.075573 20635 raft_consensus.cc:359] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37375 } }
I20251023 20:13:12.075733 20635 raft_consensus.cc:385] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:13:12.075764 20635 raft_consensus.cc:740] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49cb3c2b4f8042aea1adc37cdd2aa84b, State: Initialized, Role: FOLLOWER
I20251023 20:13:12.075855 20635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37375 } }
I20251023 20:13:12.075920 20635 raft_consensus.cc:399] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:13:12.075956 20635 raft_consensus.cc:493] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:13:12.075999 20635 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:13:12.076763 20635 raft_consensus.cc:515] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37375 } }
I20251023 20:13:12.076884 20635 leader_election.cc:304] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49cb3c2b4f8042aea1adc37cdd2aa84b; no voters:
I20251023 20:13:12.077056 20635 leader_election.cc:290] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:13:12.077113 20640 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:13:12.077279 20640 raft_consensus.cc:697] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [term 1 LEADER]: Becoming Leader. State: Replica: 49cb3c2b4f8042aea1adc37cdd2aa84b, State: Running, Role: LEADER
I20251023 20:13:12.077311 20635 sys_catalog.cc:565] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:13:12.078028 20640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37375 } }
I20251023 20:13:12.078800 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37375 } } }
I20251023 20:13:12.078809 20641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49cb3c2b4f8042aea1adc37cdd2aa84b. Latest consensus state: current_term: 1 leader_uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cb3c2b4f8042aea1adc37cdd2aa84b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37375 } } }
I20251023 20:13:12.078959 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [sys.catalog]: This master's current role is: LEADER
I20251023 20:13:12.079066 20641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b [sys.catalog]: This master's current role is: LEADER
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[20551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
W20251023 20:13:12.079850 20655 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:13:12.080308 20655 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:13:12.080384 20652 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:13:12.080561 20652 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:13:12.082458 20652 catalog_manager.cc:1357] Generated new cluster ID: a0e1cee4b67a4490b4fe79002d5be89f
I20251023 20:13:12.082516 20652 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:13:12.084204 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.073923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.9.117.254:37375)
1023 20:13:12.074120 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.074133 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.074159 (+ 26us) server_negotiation.cc:408] Connection header received
1023 20:13:12.074569 (+ 410us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.074578 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.074637 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.074804 (+ 167us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:12.075514 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.076643 (+ 1129us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.077434 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.077665 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.080780 (+ 3115us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:12.080790 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:12.080790 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:13:12.080815 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:12.082884 (+ 2069us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.083237 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.083240 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.083240 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.083291 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.083442 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.083443 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.083444 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.083649 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:12.083801 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.083923 (+ 122us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.084056 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20251023 20:13:12.121629 20652 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:13:12.122363 20652 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:13:12.128773 20652 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b: Generated new TSK 0
I20251023 20:13:12.129031 20652 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:13:12.187515 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37375
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:13:12.281566 20663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:12.281746 20663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:12.281773 20663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:12.283232 20663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:12.283279 20663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:12.283306 20663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:13:12.284937 20663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:12.285153 20663 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:12.285347 20663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:12.288029 20668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:12.288074 20671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:12.288056 20669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:12.288184 20663 server_base.cc:1047] running on GCE node
I20251023 20:13:12.288378 20663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:12.288594 20663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:12.289779 20663 hybrid_clock.cc:648] HybridClock initialized: now 1761250392289766 us; error 33 us; skew 500 ppm
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:13:12.292413 20663 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:13:12.293103 20663 webserver.cc:492] Webserver started at http://127.9.117.193:34913/ using document root <none> and password file <none>
I20251023 20:13:12.293303 20663 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:12.293344 20663 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:12.293463 20663 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:12.294452 20663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "92ac30614611488b88c9ec5eb4d1cd26"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "50c9d55ffc6ef7ba8605f721cff237c4"
server_key_iv: "4b7446eb60ceaab74751da28ac15c9a9"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.294776 20663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "92ac30614611488b88c9ec5eb4d1cd26"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "50c9d55ffc6ef7ba8605f721cff237c4"
server_key_iv: "4b7446eb60ceaab74751da28ac15c9a9"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.296159 20663 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:13:12.297001 20678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.297161 20663 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:13:12.297235 20663 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "92ac30614611488b88c9ec5eb4d1cd26"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "50c9d55ffc6ef7ba8605f721cff237c4"
server_key_iv: "4b7446eb60ceaab74751da28ac15c9a9"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.297295 20663 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:12.320822 20663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:12.322572 20663 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:12.322721 20663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:12.322993 20663 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:12.323325 20663 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:12.323359 20663 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.323397 20663 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:12.323423 20663 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.329285 20663 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:35599
I20251023 20:13:12.329351 20791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:35599 every 8 connection(s)
I20251023 20:13:12.329653 20663 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:13:12.332970 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 20663
I20251023 20:13:12.333068 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:13:12.333261 9687 external_mini_cluster.cc:1468] Setting key 7ae3ff75d644dd90ac2fdd0be5d81dee
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[20551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:13:12.338881 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.330307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:35203 (local address 127.9.117.254:37375)
1023 20:13:12.330365 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.330367 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.330523 (+ 156us) server_negotiation.cc:408] Connection header received
1023 20:13:12.331188 (+ 665us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.331190 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.331201 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.331246 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:12.332030 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.332536 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.333225 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.333422 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.336027 (+ 2605us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:12.336031 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:12.336031 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:13:12.336039 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:12.337478 (+ 1439us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.337992 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.337993 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.337994 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.338020 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.338248 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.338249 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.338249 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.338337 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:12.338385 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.338731 (+ 346us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.338814 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:13:12.339139 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.330246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37375 (local address 127.9.117.193:35203)
1023 20:13:12.330407 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.330440 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:12.331078 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:12.331305 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:12.331310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:12.331660 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:12.331961 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.331965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.332623 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.332624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:12.333133 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.333136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.333253 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.334303 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:12.334318 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:12.335926 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:12.337557 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:13:12.337564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:13:12.337567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:13:12.337888 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:13:12.338096 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:13:12.338097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:13:12.338097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:13:12.338188 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:13:12.338395 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:12.338400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:12.338582 (+ 182us) client_negotiation.cc:769] Sending connection context
1023 20:13:12.338715 (+ 133us) client_negotiation.cc:240] Negotiation successful
1023 20:13:12.338897 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20251023 20:13:12.339788 20792 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37375
I20251023 20:13:12.339882 20792 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:12.340086 20792 heartbeater.cc:507] Master 127.9.117.254:37375 requested a full tablet report, sending...
I20251023 20:13:12.340579 20599 ts_manager.cc:194] Registered new tserver with Master: 92ac30614611488b88c9ec5eb4d1cd26 (127.9.117.193:35599)
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:13:12.341245 20599 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:35203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:13:12.380049 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37375
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:13:12.463732 20799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:12.463944 20799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:12.463968 20799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:12.465410 20799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:12.465454 20799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:12.465479 20799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:13:12.467167 20799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:12.467382 20799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:12.467530 20799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:13:12.470427 20799 server_base.cc:1047] running on GCE node
W20251023 20:13:12.470336 20805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:12.470336 20804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:12.470336 20807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:12.470938 20799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:12.471166 20799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:12.472316 20799 hybrid_clock.cc:648] HybridClock initialized: now 1761250392472295 us; error 39 us; skew 500 ppm
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:13:12.475003 20799 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:13:12.475575 20799 webserver.cc:492] Webserver started at http://127.9.117.194:33059/ using document root <none> and password file <none>
I20251023 20:13:12.475777 20799 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:12.475819 20799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:12.475946 20799 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:12.477008 20799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "306e081deb0b453eb5b2830e6e93452c"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "311fead5fbbc20f2c84afba5fc1b8cde"
server_key_iv: "3f81380013b1e940df615b0df1ee2dcd"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.477326 20799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "306e081deb0b453eb5b2830e6e93452c"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "311fead5fbbc20f2c84afba5fc1b8cde"
server_key_iv: "3f81380013b1e940df615b0df1ee2dcd"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.478624 20799 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:13:12.479401 20814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.479604 20799 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:13:12.479682 20799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "306e081deb0b453eb5b2830e6e93452c"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "311fead5fbbc20f2c84afba5fc1b8cde"
server_key_iv: "3f81380013b1e940df615b0df1ee2dcd"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.479751 20799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:12.493072 20799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:12.494956 20799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:12.495105 20799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:12.495347 20799 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:12.495688 20799 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:12.495726 20799 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.495764 20799 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:12.495792 20799 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.501978 20799 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:33095
I20251023 20:13:12.502061 20927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:33095 every 8 connection(s)
I20251023 20:13:12.502362 20799 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:13:12.505106 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 20799
I20251023 20:13:12.505230 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:13:12.505414 9687 external_mini_cluster.cc:1468] Setting key 1b35c0ffd1960ad8e260d18fd631a6f4
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[20551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:13:12.512276 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.503091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:33443 (local address 127.9.117.254:37375)
1023 20:13:12.503169 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.503173 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.503314 (+ 141us) server_negotiation.cc:408] Connection header received
1023 20:13:12.503860 (+ 546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.503861 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.503876 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.503932 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:12.504758 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.505331 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.506086 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.506264 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.509047 (+ 2783us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:12.509052 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:12.509053 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:13:12.509061 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:12.510752 (+ 1691us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.511301 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.511302 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.511302 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.511324 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.511554 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.511555 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.511555 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.511663 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:12.511717 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.512114 (+ 397us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.512204 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251023 20:13:12.512511 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.503022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37375 (local address 127.9.117.194:33443)
1023 20:13:12.503200 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.503231 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:12.503754 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:12.503998 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:12.504002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:12.504335 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:12.504624 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.504627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.505425 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.505427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:12.505996 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.505999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.506158 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.507225 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:12.507245 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:12.508934 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:12.510860 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:13:12.510869 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:13:12.510873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:13:12.511217 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:13:12.511387 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:13:12.511388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:13:12.511389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:13:12.511488 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:13:12.511725 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:12.511730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:12.511922 (+ 192us) client_negotiation.cc:769] Sending connection context
1023 20:13:12.512071 (+ 149us) client_negotiation.cc:240] Negotiation successful
1023 20:13:12.512242 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20251023 20:13:12.513121 20928 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37375
I20251023 20:13:12.513232 20928 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:12.513453 20928 heartbeater.cc:507] Master 127.9.117.254:37375 requested a full tablet report, sending...
I20251023 20:13:12.513875 20599 ts_manager.cc:194] Registered new tserver with Master: 306e081deb0b453eb5b2830e6e93452c (127.9.117.194:33095)
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
I20251023 20:13:12.514429 20599 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:33443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:13:12.556577 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37375
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:13:12.639128 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:13:12.639331 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:13:12.639358 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:13:12.640879 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:13:12.640926 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:13:12.640954 20935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:13:12.642609 20935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:13:12.642841 20935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:13:12.643010 20935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:13:12.645745 20941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:12.645751 20940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:13:12.645812 20943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:13:12.645978 20935 server_base.cc:1047] running on GCE node
I20251023 20:13:12.646160 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:13:12.646349 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:13:12.647493 20935 hybrid_clock.cc:648] HybridClock initialized: now 1761250392647471 us; error 36 us; skew 500 ppm
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:13:12.650239 20935 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:13:12.650800 20935 webserver.cc:492] Webserver started at http://127.9.117.195:35663/ using document root <none> and password file <none>
I20251023 20:13:12.651013 20935 fs_manager.cc:362] Metadata directory not provided
I20251023 20:13:12.651058 20935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:13:12.651182 20935 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:13:12.652194 20935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "33df6d7e8f2f4112a047f0ed3c124774"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "f40d91522a539bf0590fd1b9597de01b"
server_key_iv: "116125118ec35190b3f6a767413aa184"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.652513 20935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "33df6d7e8f2f4112a047f0ed3c124774"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "f40d91522a539bf0590fd1b9597de01b"
server_key_iv: "116125118ec35190b3f6a767413aa184"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.653913 20935 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:13:12.654700 20950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.654884 20935 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:13:12.654966 20935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "33df6d7e8f2f4112a047f0ed3c124774"
format_stamp: "Formatted at 2025-10-23 20:13:12 on dist-test-slave-x62p"
server_key: "f40d91522a539bf0590fd1b9597de01b"
server_key_iv: "116125118ec35190b3f6a767413aa184"
server_key_version: "encryptionkey@0"
I20251023 20:13:12.655035 20935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:13:12.666020 20935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:13:12.668111 20935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:13:12.668257 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:13:12.668498 20935 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:13:12.668895 20935 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:13:12.668960 20935 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.668995 20935 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:13:12.669013 20935 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:13:12.675127 20935 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:39395
I20251023 20:13:12.675182 21063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:39395 every 8 connection(s)
I20251023 20:13:12.675482 20935 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[20551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:13:12.681485 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 20935
I20251023 20:13:12.681569 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:13:12.681762 9687 external_mini_cluster.cc:1468] Setting key de27bb780079b1da7325fb937357ca31
I20251023 20:13:12.684500 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.676176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:40135 (local address 127.9.117.254:37375)
1023 20:13:12.676255 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.676257 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.676396 (+ 139us) server_negotiation.cc:408] Connection header received
1023 20:13:12.676986 (+ 590us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.676988 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.677001 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.677060 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:12.677886 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.678398 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.678957 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.679159 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.681613 (+ 2454us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:12.681618 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:12.681618 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:13:12.681626 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:12.683239 (+ 1613us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.683651 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.683652 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.683652 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.683677 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.683858 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.683859 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.683859 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.683969 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:12.684041 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.684357 (+ 316us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.684433 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:13:12.684839 21066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.676122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37375 (local address 127.9.117.195:40135)
1023 20:13:12.676291 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.676317 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:12.676882 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:12.677120 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:12.677126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:12.677459 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:13:12.677783 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.677787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.678480 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.678481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:12.678870 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.678871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.679007 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.679966 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:13:12.679982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:13:12.681517 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:13:12.683322 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:13:12.683332 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:13:12.683335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:13:12.683555 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:13:12.683734 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:13:12.683735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:13:12.683736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:13:12.683801 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:13:12.684051 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:13:12.684056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:13:12.684196 (+ 140us) client_negotiation.cc:769] Sending connection context
1023 20:13:12.684347 (+ 151us) client_negotiation.cc:240] Negotiation successful
1023 20:13:12.684508 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20251023 20:13:12.685369 21064 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37375
I20251023 20:13:12.685462 21064 heartbeater.cc:461] Registering TS with master...
I20251023 20:13:12.685694 21064 heartbeater.cc:507] Master 127.9.117.254:37375 requested a full tablet report, sending...
I20251023 20:13:12.686066 20599 ts_manager.cc:194] Registered new tserver with Master: 33df6d7e8f2f4112a047f0ed3c124774 (127.9.117.195:39395)
I20251023 20:13:12.686441 20599 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:40135
I20251023 20:13:12.693868 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:13:12.701436 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.695369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.9.117.254:37375)
1023 20:13:12.695436 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.695438 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.695586 (+ 148us) server_negotiation.cc:408] Connection header received
1023 20:13:12.695617 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.695618 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.695629 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.695685 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:13:12.696402 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.696992 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.697582 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.697702 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.698787 (+ 1085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:13:12.698791 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:13:12.698791 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:13:12.698797 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:13:12.700109 (+ 1312us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.700535 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.700537 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.700537 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.700566 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:13:12.700828 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:13:12.700829 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:13:12.700829 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:13:12.700972 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:13:12.701062 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.701264 (+ 202us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.701346 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:13:12.703284 20599 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:13:12.704654 20599 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:13:12.711115 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.707255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:33095 (local address 127.0.0.1:45042)
1023 20:13:12.707657 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.707692 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:12.707794 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:12.708220 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:12.708221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:12.708225 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:13:12.708507 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.708509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.709972 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.709973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:12.710756 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.710759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.710833 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.710901 (+ 68us) client_negotiation.cc:769] Sending connection context
1023 20:13:12.710958 (+ 57us) client_negotiation.cc:240] Negotiation successful
1023 20:13:12.711012 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":81,"threads_started":1}
I20251023 20:13:12.711328 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.707460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.9.117.195:39395)
1023 20:13:12.707788 (+ 328us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.707800 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.707966 (+ 166us) server_negotiation.cc:408] Connection header received
1023 20:13:12.708032 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.708035 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.708054 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.708126 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:13:12.708507 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.709313 (+ 806us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.710381 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.710958 (+ 577us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.711028 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.711162 (+ 134us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.711222 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":86,"threads_started":1}
I20251023 20:13:12.711577 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.707258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:39395 (local address 127.0.0.1:51978)
1023 20:13:12.707890 (+ 632us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.707898 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:12.707949 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:12.708213 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:12.708214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:12.708217 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:13:12.708438 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.708440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.709408 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.709409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:12.710242 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.710246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.711061 (+ 815us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.711064 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:13:12.711463 (+ 399us) client_negotiation.cc:240] Negotiation successful
1023 20:13:12.711517 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":5,"thread_start_us":114,"threads_started":1}
I20251023 20:13:12.711674 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.707466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.9.117.194:33095)
1023 20:13:12.707666 (+ 200us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.707675 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.707887 (+ 212us) server_negotiation.cc:408] Connection header received
1023 20:13:12.707935 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.707937 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.707957 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.708029 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:13:12.708606 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.709830 (+ 1224us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.710879 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.711483 (+ 604us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.711526 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.711553 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.711587 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20251023 20:13:12.712301 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.707694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.9.117.193:35599)
1023 20:13:12.707896 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.707904 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.707915 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:13:12.707957 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.707959 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.707982 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.708052 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:13:12.708498 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.709830 (+ 1332us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.711533 (+ 1703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.712144 (+ 611us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.712178 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.712208 (+ 30us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.712241 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20251023 20:13:12.712324 20862 tablet_service.cc:1505] Processing CreateTablet for tablet dbe991d865894fd49d8dc9e4cf466852 (DEFAULT_TABLE table=test-table [id=e13ac2bddcc54f64829ccc7a56a162f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:13:12.712298 20998 tablet_service.cc:1505] Processing CreateTablet for tablet dbe991d865894fd49d8dc9e4cf466852 (DEFAULT_TABLE table=test-table [id=e13ac2bddcc54f64829ccc7a56a162f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:13:12.712461 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.707258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:35599 (local address 127.0.0.1:45306)
1023 20:13:12.707722 (+ 464us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.707735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:12.707805 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:12.708068 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:12.708070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:12.708073 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:13:12.708394 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.708397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.709936 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.709937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:12.710866 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.710868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.710949 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.710953 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:13:12.710984 (+ 31us) client_negotiation.cc:240] Negotiation successful
1023 20:13:12.711028 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":124,"threads_started":1}
I20251023 20:13:12.712682 20998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbe991d865894fd49d8dc9e4cf466852. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:13:12.712680 20862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbe991d865894fd49d8dc9e4cf466852. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:13:12.713094 20726 tablet_service.cc:1505] Processing CreateTablet for tablet dbe991d865894fd49d8dc9e4cf466852 (DEFAULT_TABLE table=test-table [id=e13ac2bddcc54f64829ccc7a56a162f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:13:12.713332 20726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbe991d865894fd49d8dc9e4cf466852. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:13:12.715642 21081 tablet_bootstrap.cc:492] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26: Bootstrap starting.
I20251023 20:13:12.716167 21082 tablet_bootstrap.cc:492] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: Bootstrap starting.
I20251023 20:13:12.716334 21081 tablet_bootstrap.cc:654] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26: Neither blocks nor log segments found. Creating new log.
I20251023 20:13:12.716665 21081 log.cc:826] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26: Log is configured to *not* fsync() on all Append() calls
I20251023 20:13:12.716784 21082 tablet_bootstrap.cc:654] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: Neither blocks nor log segments found. Creating new log.
I20251023 20:13:12.716928 21083 tablet_bootstrap.cc:492] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Bootstrap starting.
I20251023 20:13:12.717118 21082 log.cc:826] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: Log is configured to *not* fsync() on all Append() calls
I20251023 20:13:12.717463 21083 tablet_bootstrap.cc:654] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Neither blocks nor log segments found. Creating new log.
I20251023 20:13:12.717696 21081 tablet_bootstrap.cc:492] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26: No bootstrap required, opened a new log
I20251023 20:13:12.717774 21081 ts_tablet_manager.cc:1403] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:13:12.717764 21083 log.cc:826] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Log is configured to *not* fsync() on all Append() calls
I20251023 20:13:12.717829 21082 tablet_bootstrap.cc:492] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: No bootstrap required, opened a new log
I20251023 20:13:12.717907 21082 ts_tablet_manager.cc:1403] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:13:12.718538 21083 tablet_bootstrap.cc:492] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: No bootstrap required, opened a new log
I20251023 20:13:12.718621 21083 ts_tablet_manager.cc:1403] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:13:12.719578 21081 raft_consensus.cc:359] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.719559 21082 raft_consensus.cc:359] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.719707 21082 raft_consensus.cc:385] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:13:12.719707 21081 raft_consensus.cc:385] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:13:12.719739 21082 raft_consensus.cc:740] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33df6d7e8f2f4112a047f0ed3c124774, State: Initialized, Role: FOLLOWER
I20251023 20:13:12.719743 21081 raft_consensus.cc:740] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ac30614611488b88c9ec5eb4d1cd26, State: Initialized, Role: FOLLOWER
I20251023 20:13:12.719872 21081 consensus_queue.cc:260] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.719874 21082 consensus_queue.cc:260] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.720182 20792 heartbeater.cc:499] Master 127.9.117.254:37375 was elected leader, sending a full tablet report...
I20251023 20:13:12.720180 21081 ts_tablet_manager.cc:1434] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251023 20:13:12.720413 21064 heartbeater.cc:499] Master 127.9.117.254:37375 was elected leader, sending a full tablet report...
I20251023 20:13:12.720484 21083 raft_consensus.cc:359] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.720628 21083 raft_consensus.cc:385] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:13:12.720659 21083 raft_consensus.cc:740] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306e081deb0b453eb5b2830e6e93452c, State: Initialized, Role: FOLLOWER
I20251023 20:13:12.720772 21083 consensus_queue.cc:260] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.720405 21082 ts_tablet_manager.cc:1434] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251023 20:13:12.721060 21083 ts_tablet_manager.cc:1434] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:13:12.721319 20928 heartbeater.cc:499] Master 127.9.117.254:37375 was elected leader, sending a full tablet report...
I20251023 20:13:12.731031 21089 raft_consensus.cc:493] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:13:12.731158 21089 raft_consensus.cc:515] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.731487 21089 leader_election.cc:290] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33df6d7e8f2f4112a047f0ed3c124774 (127.9.117.195:39395), 92ac30614611488b88c9ec5eb4d1cd26 (127.9.117.193:35599)
I20251023 20:13:12.733793 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.731644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:39395 (local address 127.9.117.194:36363)
1023 20:13:12.731703 (+ 59us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.731717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:12.731774 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:12.731955 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:12.731957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:12.731958 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:13:12.732149 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.732150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.732993 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.732995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:12.733561 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.733563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.733637 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.733643 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:13:12.733676 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:13:12.733714 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251023 20:13:12.733968 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.731685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:35599 (local address 127.9.117.194:37237)
1023 20:13:12.731879 (+ 194us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.731886 (+ 7us) client_negotiation.cc:174] Beginning negotiation
1023 20:13:12.731928 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:13:12.732086 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:13:12.732087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:13:12.732089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:13:12.732317 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.732318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.733146 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.733146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:13:12.733739 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:13:12.733740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.733814 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.733817 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:13:12.733848 (+ 31us) client_negotiation.cc:240] Negotiation successful
1023 20:13:12.733886 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":120,"threads_started":1}
I20251023 20:13:12.734280 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.731739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:36363 (local address 127.9.117.195:39395)
1023 20:13:12.731803 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.731805 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.731818 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:13:12.731848 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.731850 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.731861 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.731930 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:13:12.732224 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.732911 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.733652 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.734177 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.734186 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.734207 (+ 21us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.734231 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251023 20:13:12.734347 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.731887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:37237 (local address 127.9.117.193:35599)
1023 20:13:12.731973 (+ 86us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.731975 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.731984 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:13:12.732015 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.732016 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.732025 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.732077 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:13:12.732388 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.733037 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.733824 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.734246 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.734254 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.734274 (+ 20us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.734298 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251023 20:13:12.734611 21018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbe991d865894fd49d8dc9e4cf466852" candidate_uuid: "306e081deb0b453eb5b2830e6e93452c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33df6d7e8f2f4112a047f0ed3c124774" is_pre_election: true
I20251023 20:13:12.734755 20746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbe991d865894fd49d8dc9e4cf466852" candidate_uuid: "306e081deb0b453eb5b2830e6e93452c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ac30614611488b88c9ec5eb4d1cd26" is_pre_election: true
I20251023 20:13:12.734863 21018 raft_consensus.cc:2468] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 306e081deb0b453eb5b2830e6e93452c in term 0.
I20251023 20:13:12.734905 20746 raft_consensus.cc:2468] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 306e081deb0b453eb5b2830e6e93452c in term 0.
I20251023 20:13:12.735165 20815 leader_election.cc:304] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306e081deb0b453eb5b2830e6e93452c, 92ac30614611488b88c9ec5eb4d1cd26; no voters:
I20251023 20:13:12.735342 21089 raft_consensus.cc:2804] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:13:12.735419 21089 raft_consensus.cc:493] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:13:12.735456 21089 raft_consensus.cc:3060] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:13:12.736202 21089 raft_consensus.cc:515] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.736366 21089 leader_election.cc:290] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [CANDIDATE]: Term 1 election: Requested vote from peers 33df6d7e8f2f4112a047f0ed3c124774 (127.9.117.195:39395), 92ac30614611488b88c9ec5eb4d1cd26 (127.9.117.193:35599)
I20251023 20:13:12.736595 21018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbe991d865894fd49d8dc9e4cf466852" candidate_uuid: "306e081deb0b453eb5b2830e6e93452c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33df6d7e8f2f4112a047f0ed3c124774"
I20251023 20:13:12.736568 20746 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbe991d865894fd49d8dc9e4cf466852" candidate_uuid: "306e081deb0b453eb5b2830e6e93452c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ac30614611488b88c9ec5eb4d1cd26"
I20251023 20:13:12.736676 20746 raft_consensus.cc:3060] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:13:12.736685 21018 raft_consensus.cc:3060] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:13:12.737473 21018 raft_consensus.cc:2468] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306e081deb0b453eb5b2830e6e93452c in term 1.
I20251023 20:13:12.737471 20746 raft_consensus.cc:2468] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306e081deb0b453eb5b2830e6e93452c in term 1.
I20251023 20:13:12.737680 20815 leader_election.cc:304] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306e081deb0b453eb5b2830e6e93452c, 92ac30614611488b88c9ec5eb4d1cd26; no voters:
I20251023 20:13:12.737792 21089 raft_consensus.cc:2804] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:13:12.737962 21089 raft_consensus.cc:697] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 1 LEADER]: Becoming Leader. State: Replica: 306e081deb0b453eb5b2830e6e93452c, State: Running, Role: LEADER
I20251023 20:13:12.738065 21089 consensus_queue.cc:237] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } }
I20251023 20:13:12.738910 20599 catalog_manager.cc:5649] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c reported cstate change: term changed from 0 to 1, leader changed from <none> to 306e081deb0b453eb5b2830e6e93452c (127.9.117.194). New cstate: current_term: 1 leader_uuid: "306e081deb0b453eb5b2830e6e93452c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306e081deb0b453eb5b2830e6e93452c" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33095 } health_report { overall_health: HEALTHY } } }
I20251023 20:13:12.751571 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.749239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.9.117.194:33095)
1023 20:13:12.749288 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:13:12.749290 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:13:12.749298 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:13:12.749340 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:13:12.749341 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:13:12.749359 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:13:12.749414 (+ 55us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:13:12.749653 (+ 239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.750154 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:13:12.750912 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:13:12.751072 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:13:12.751121 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:13:12.751232 (+ 111us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:13:12.751311 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1023 20:13:12.751467 (+ 156us) server_negotiation.cc:300] Negotiation successful
1023 20:13:12.751501 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20251023 20:13:12.752869 20929 tablet.cc:2378] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:13:12.753588 21018 raft_consensus.cc:1275] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 1 FOLLOWER]: Refusing update from remote peer 306e081deb0b453eb5b2830e6e93452c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:13:12.753588 20746 raft_consensus.cc:1275] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [term 1 FOLLOWER]: Refusing update from remote peer 306e081deb0b453eb5b2830e6e93452c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:13:12.753952 21089 consensus_queue.cc:1048] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [LEADER]: Connected to new peer: Peer: permanent_uuid: "33df6d7e8f2f4112a047f0ed3c124774" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:13:12.754101 21091 consensus_queue.cc:1048] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ac30614611488b88c9ec5eb4d1cd26" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 35599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:13:12.756528 21094 mvcc.cc:204] Tried to move back new op lower bound from 7214081608715862016 to 7214081608655798272. Current Snapshot: MvccSnapshot[applied={T|T < 7214081608715862016}]
I20251023 20:13:12.757535 21095 mvcc.cc:204] Tried to move back new op lower bound from 7214081608715862016 to 7214081608655798272. Current Snapshot: MvccSnapshot[applied={T|T < 7214081608715862016 or (T in {7214081608715862016})}]
I20251023 20:13:12.757668 21096 mvcc.cc:204] Tried to move back new op lower bound from 7214081608715862016 to 7214081608655798272. Current Snapshot: MvccSnapshot[applied={T|T < 7214081608715862016}]
I20251023 20:13:12.759342 20599 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52582:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:13:12.759435 20599 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52582:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:13:12.760274 20599 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 49cb3c2b4f8042aea1adc37cdd2aa84b: Sending DeleteTablet for 3 replicas of tablet dbe991d865894fd49d8dc9e4cf466852
I20251023 20:13:12.760632 20726 tablet_service.cc:1552] Processing DeleteTablet for tablet dbe991d865894fd49d8dc9e4cf466852 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:13:12 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:45306
I20251023 20:13:12.760649 20998 tablet_service.cc:1552] Processing DeleteTablet for tablet dbe991d865894fd49d8dc9e4cf466852 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:13:12 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:51978
I20251023 20:13:12.760649 20862 tablet_service.cc:1552] Processing DeleteTablet for tablet dbe991d865894fd49d8dc9e4cf466852 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:13:12 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:45042
I20251023 20:13:12.760951 21106 tablet_replica.cc:333] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: stopping tablet replica
I20251023 20:13:12.761046 21106 raft_consensus.cc:2243] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:13:12.761183 21104 tablet_replica.cc:333] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26: stopping tablet replica
I20251023 20:13:12.761281 21104 raft_consensus.cc:2243] T dbe991d865894fd49d8dc9e4cf466852 P 92ac30614611488b88c9ec5eb4d1cd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:13:12.761381 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 20663
I20251023 20:13:12.764946 21106 raft_consensus.cc:2272] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:13:12.765199 21105 tablet_replica.cc:333] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: stopping tablet replica
I20251023 20:13:12.765311 21105 raft_consensus.cc:2243] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:13:12.765432 21105 raft_consensus.cc:2272] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:13:12.765698 21106 ts_tablet_manager.cc:1916] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:13:12.765931 21105 ts_tablet_manager.cc:1916] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:13:12.767604 21106 ts_tablet_manager.cc:1929] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:13:12.767777 21106 log.cc:1199] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/dbe991d865894fd49d8dc9e4cf466852
I20251023 20:13:12.768019 21105 ts_tablet_manager.cc:1929] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:13:12.768105 21105 log.cc:1199] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/dbe991d865894fd49d8dc9e4cf466852
I20251023 20:13:12.768291 21106 ts_tablet_manager.cc:1950] T dbe991d865894fd49d8dc9e4cf466852 P 306e081deb0b453eb5b2830e6e93452c: Deleting consensus metadata
I20251023 20:13:12.768409 21105 ts_tablet_manager.cc:1950] T dbe991d865894fd49d8dc9e4cf466852 P 33df6d7e8f2f4112a047f0ed3c124774: Deleting consensus metadata
W20251023 20:13:12.768597 20583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:35599 (error 108)
I20251023 20:13:12.768929 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 20799
I20251023 20:13:12.769083 20585 catalog_manager.cc:4985] TS 33df6d7e8f2f4112a047f0ed3c124774 (127.9.117.195:39395): tablet dbe991d865894fd49d8dc9e4cf466852 (table test-table [id=e13ac2bddcc54f64829ccc7a56a162f5]) successfully deleted
I20251023 20:13:12.769568 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.769446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:35599 (local address 127.0.0.1:45320)
1023 20:13:12.769497 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.769537 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20251023 20:13:12.769688 20583 catalog_manager.cc:4712] TS 92ac30614611488b88c9ec5eb4d1cd26 (127.9.117.193:35599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbe991d865894fd49d8dc9e4cf466852: Network error: Client connection negotiation failed: client connection to 127.9.117.193:35599: connect: Connection refused (error 111)
I20251023 20:13:12.774344 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 20935
I20251023 20:13:12.774518 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:13:12.774361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:33095 (local address 127.0.0.1:45052)
1023 20:13:12.774440 (+ 79us) negotiation.cc:107] Waiting for socket to connect
1023 20:13:12.774473 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.194:33095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20251023 20:13:12.779677 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 20567
2025-10-23T20:13:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2987 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[21110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[21110](info): set up 2 sockets
Oct 23 20:13:12 dist-test-slave-x62p krb5kdc[21110](info): commencing operation
krb5kdc: starting...
W20251023 20:13:14.793777 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.993s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:13:14 dist-test-slave-x62p krb5kdc[21110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:13:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:13:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.117.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.117.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/rangeradmin_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/rangeradmin_127.9.117.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.117.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.117.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/rangerlookup_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/rangerlookup_127.9.117.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.117.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/HTTP_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/HTTP_127.9.117.212@KRBTEST.COM.keytab.
I20251023 20:13:14.959633 9687 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskut_hJr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/postgres -l logfile start
2025-10-23 20:13:17.638 UTC [21141] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-23 20:13:17.638 UTC [21141] LOG: listening on IPv4 address "127.9.117.212", port 46615
2025-10-23 20:13:17.643 UTC [21141] LOG: listening on Unix socket "/tmp/.s.PGSQL.46615"
2025-10-23 20:13:17.648 UTC [21146] LOG: database system was shut down at 2025-10-23 20:13:16 UTC
2025-10-23 20:13:17.653 UTC [21141] LOG: database system is ready to accept connections
I20251023 20:13:19.561535 9687 mini_postgres.cc:96] Postgres bound to 46615
2025-10-23 20:13:19.568 UTC [21152] FATAL: database "slave" does not exist
127.9.117.212:46615 - accepting connections
I20251023 20:13:19.568862 9687 mini_ranger.cc:161] Starting Ranger...
I20251023 20:13:19.589166 9687 mini_ranger.cc:84] Created miniranger Postgres user
I20251023 20:13:19.670589 9687 mini_ranger.cc:87] Created ranger Postgres database
I20251023 20:13:19.670671 9687 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-admin
2025-10-23 20:13:20,394 [I] DB FLAVOR :POSTGRES
2025-10-23 20:13:20,394 [I] --------- Verifying Ranger DB connection ---------
2025-10-23 20:13:20,394 [I] Checking connection..
2025-10-23 20:13:20,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-23 20:13:20,805 [I] Checking connection passed.
2025-10-23 20:13:20,805 [I] --------- Verifying version history table ---------
2025-10-23 20:13:20,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-23 20:13:21,130 [I] Table x_db_version_h does not exist in database ranger
2025-10-23 20:13:21,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-23 20:13:21,471 [I] Table x_db_version_h does not exist in database ranger
2025-10-23 20:13:21,472 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-23 20:13:21,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-23 20:13:21.818 UTC [21219] WARNING: there is no transaction in progress
2025-10-23 20:13:21,824 [I] create_dbversion_catalog.sql file imported successfully
2025-10-23 20:13:21,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-23 20:13:22,165 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-23 20:13:22,165 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-23 20:13:22,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-23 20:13:22,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-23 20:13:22,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-x62p.c.gcp-upstream.internal','N') ;"
2025-10-23 20:13:23,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-23 20:13:23,510 [I] Table x_portal_user does not exist in database ranger
2025-10-23 20:13:23,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-23 20:13:23,862 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-23 20:13:23,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-23 20:13:24,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-23 20:13:24,556 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-23 20:13:24,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-23 20:13:25.451 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.463 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.477 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.488 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.504 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.517 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.569 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.574 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.585 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.595 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.606 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.616 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.625 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.630 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.635 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:25.639 UTC [21336] WARNING: there is no transaction in progress
2025-10-23 20:13:26,063 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-23 20:13:26,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-23 20:13:26,399 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-23 20:13:26,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-23 20:13:26,744 [I] Table x_portal_user already exists in database 'ranger'
2025-10-23 20:13:26,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-23 20:13:27,094 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-23 20:13:27,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-23 20:13:27,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-23 20:13:27,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-x62p.c.gcp-upstream.internal';"
2025-10-23 20:13:28,151 [I] CORE_DB_SCHEMA import status has been updated
2025-10-23 20:13:28,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-23 20:13:28,495 [I] DB_PATCHES have already been applied
I20251023 20:13:28.501425 9687 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-admin:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskut_hJr/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251023 20:13:28.501497 9687 mini_ranger.cc:193] Using host: 127.9.117.212
I20251023 20:13:28.504309 9687 mini_ranger.cc:227] Ranger admin URL: http://127.9.117.212:48115
Oct 23, 2025 8:13:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 23, 2025 8:13:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 23, 2025 8:13:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-admin/ews/webapp
Oct 23, 2025 8:13:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 23, 2025 8:13:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 23, 2025 8:13:29 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 23, 2025 8:13:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-admin/ews/webapp].
Oct 23, 2025 8:13:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 23, 2025 8:13:29 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-48115"]
Oct 23, 2025 8:13:29 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 23, 2025 8:13:29 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 23, 2025 8:13:30 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 23, 2025 8:13:30 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 23, 2025 8:13:30 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20251023 20:13:30.712193 9687 mini_ranger.cc:160] Time spent starting Ranger: real 11.143s user 0.002s sys 0.008s
Oct 23, 2025 8:13:31 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 23, 2025 8:13:31 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 23, 2025 8:13:31 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 23, 2025 8:13:31 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Oct 23, 2025 8:13:53 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 23, 2025 8:13:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 23, 2025 8:13:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 23, 2025 8:13:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 23, 2025 8:13:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 23, 2025 8:13:54 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-48115"]
I20251023 20:13:55.485654 9687 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.117.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.117.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/rangerkms_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/rangerkms_127.9.117.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.117.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.117.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/HTTP_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.117.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/HTTP_127.9.117.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 23 20:13:55 dist-test-slave-x62p krb5kdc[21110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250435, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251023 20:13:55.568995 9687 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251023 20:13:55.580176 9687 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251023 20:13:55.703724 9687 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251023 20:13:55.703822 9687 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms
I20251023 20:13:55.703840 9687 mini_ranger_kms.cc:227] Using postgres at 127.9.117.212:46615
2025-10-23 20:13:56,403 [I] DB FLAVOR :POSTGRES
2025-10-23 20:13:56,403 [I] --------- Verifying Ranger DB connection ---------
2025-10-23 20:13:56,403 [I] Checking connection
2025-10-23 20:13:56,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-23 20:13:56,757 [I] connection success
2025-10-23 20:13:56,757 [I] --------- Verifying Ranger DB tables ---------
2025-10-23 20:13:56,758 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-23 20:13:56,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-23 20:13:57,094 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-23 20:13:57,094 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-23 20:13:57,094 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-23 20:13:57,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:46615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-23 20:13:57,454 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251023 20:13:57.992606 9687 mini_ranger_kms.cc:310] Created kms service
I20251023 20:13:58.243256 9687 mini_ranger_kms.cc:326] Created kudu user
I20251023 20:13:58.299147 9687 mini_ranger_kms.cc:343] Created rangerkms user
Oct 23 20:13:58 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250435, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:13:58.755407 9687 mini_ranger_kms.cc:384] Added ranger policy
I20251023 20:13:58.755641 9687 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms:/tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251023 20:13:58.755678 9687 mini_ranger_kms.cc:240] Using host: 127.9.117.212
I20251023 20:13:58.759260 9687 mini_ranger_kms.cc:276] Ranger KMS PID: 21528
I20251023 20:13:58.759392 9687 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.117.212:33047
Oct 23, 2025 8:13:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 23, 2025 8:13:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 23, 2025 8:13:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 23, 2025 8:13:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 23, 2025 8:14:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-kms/ews/webapp].
Oct 23, 2025 8:14:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 23, 2025 8:14:00 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-33047"]
Oct 23, 2025 8:14:00 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 23, 2025 8:14:00 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 23, 2025 8:14:00 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 23, 2025 8:14:00 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
I20251023 20:14:01.051332 9687 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.482s user 0.000s sys 0.300s
I20251023 20:14:01.051472 9687 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251023 20:14:01.051546 9687 mini_ranger_kms.cc:401] 127.9.117.212:33047/kms/v1/keys
Oct 23 20:14:01 dist-test-slave-x62p krb5kdc[21110](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761250441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.117.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Oct 23 20:14:01 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.117.212@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
Oct 23 20:14:02 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.117.212@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
Oct 23 20:14:04 dist-test-slave-x62p krb5kdc[21110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:14:04.792006 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:34405
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:34405
--ranger_config_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:33047/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:14:04.877434 21581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:14:04.877662 21581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:14:04.877689 21581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:14:04.879348 21581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:14:04.879413 21581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:14:04.879428 21581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:14:04.879441 21581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:14:04.879454 21581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:14:04.881506 21581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.117.212:33047/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:34405
--ranger_config_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:34405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:14:04.881770 21581 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:14:04.881997 21581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:14:04.885413 21586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:14:04.885645 21589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:14:04.885439 21581 server_base.cc:1047] running on GCE node
W20251023 20:14:04.885416 21587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:14:04.886144 21581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:14:04.886492 21581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:14:04.887693 21581 hybrid_clock.cc:648] HybridClock initialized: now 1761250444887671 us; error 41 us; skew 500 ppm
Oct 23 20:14:04 dist-test-slave-x62p krb5kdc[21110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250444, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:14:04.890668 21581 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:14:04.891417 21581 webserver.cc:492] Webserver started at http://127.9.117.254:36543/ using document root <none> and password file <none>
I20251023 20:14:04.891640 21581 fs_manager.cc:362] Metadata directory not provided
I20251023 20:14:04.891685 21581 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:14:04.891804 21581 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 23 20:14:04 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250444, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:14:04.942462 21581 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "1ba41f83d6e240b0bdd7979fe7afae71"
format_stamp: "Formatted at 2025-10-23 20:14:04 on dist-test-slave-x62p"
server_key: "d9ddde4f0121355ff93c8a350ede88a7"
server_key_iv: "d34384c43cf238bb6618ab0f86d862e9"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:04.942958 21581 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "1ba41f83d6e240b0bdd7979fe7afae71"
format_stamp: "Formatted at 2025-10-23 20:14:04 on dist-test-slave-x62p"
server_key: "d9ddde4f0121355ff93c8a350ede88a7"
server_key_iv: "d34384c43cf238bb6618ab0f86d862e9"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:04.944860 21581 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20251023 20:14:04.960781 21596 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:04.960994 21581 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:14:04.961072 21581 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "1ba41f83d6e240b0bdd7979fe7afae71"
format_stamp: "Formatted at 2025-10-23 20:14:04 on dist-test-slave-x62p"
server_key: "d9ddde4f0121355ff93c8a350ede88a7"
server_key_iv: "d34384c43cf238bb6618ab0f86d862e9"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:04.961172 21581 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:14:04.974437 21581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:14:04.976435 21581 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:14:04.976577 21581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:14:04.981573 21581 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:34405
I20251023 20:14:04.981818 21648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:34405 every 8 connection(s)
I20251023 20:14:04.982033 21581 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:14:04.982761 21649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:14:04.985870 21649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71: Bootstrap starting.
I20251023 20:14:04.986652 21649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71: Neither blocks nor log segments found. Creating new log.
I20251023 20:14:04.986946 21649 log.cc:826] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71: Log is configured to *not* fsync() on all Append() calls
I20251023 20:14:04.987823 21649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71: No bootstrap required, opened a new log
I20251023 20:14:04.988183 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 21581
I20251023 20:14:04.988301 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:14:04.988488 9687 external_mini_cluster.cc:1468] Setting key f3f7f4652b0b1f75d316a01f24f4a28d
I20251023 20:14:04.989971 21649 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba41f83d6e240b0bdd7979fe7afae71" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34405 } }
I20251023 20:14:04.990222 21649 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:14:04.990272 21649 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba41f83d6e240b0bdd7979fe7afae71, State: Initialized, Role: FOLLOWER
I20251023 20:14:04.990389 21649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba41f83d6e240b0bdd7979fe7afae71" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34405 } }
I20251023 20:14:04.990453 21649 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:14:04.990504 21649 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:14:04.990568 21649 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:14:04.991328 21649 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba41f83d6e240b0bdd7979fe7afae71" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34405 } }
I20251023 20:14:04.991446 21649 leader_election.cc:304] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba41f83d6e240b0bdd7979fe7afae71; no voters:
I20251023 20:14:04.991645 21649 leader_election.cc:290] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:14:04.991714 21654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:14:04.991922 21654 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba41f83d6e240b0bdd7979fe7afae71, State: Running, Role: LEADER
I20251023 20:14:04.991958 21649 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:14:04.992079 21654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba41f83d6e240b0bdd7979fe7afae71" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34405 } }
I20251023 20:14:04.992965 21655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba41f83d6e240b0bdd7979fe7afae71. Latest consensus state: current_term: 1 leader_uuid: "1ba41f83d6e240b0bdd7979fe7afae71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba41f83d6e240b0bdd7979fe7afae71" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34405 } } }
I20251023 20:14:04.993130 21655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [sys.catalog]: This master's current role is: LEADER
I20251023 20:14:04.993739 21649 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251023 20:14:04.994117 21668 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:14:04.994717 21668 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 23 20:14:04 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:14:04.996273 21654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba41f83d6e240b0bdd7979fe7afae71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba41f83d6e240b0bdd7979fe7afae71" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34405 } } }
I20251023 20:14:04.996444 21654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71 [sys.catalog]: This master's current role is: LEADER
I20251023 20:14:05.002346 21668 catalog_manager.cc:1357] Generated new cluster ID: 45157d1572ae4549abca9d7821a80cc5
I20251023 20:14:05.002514 21668 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:14:05.004073 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:04.989126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.9.117.254:34405)
1023 20:14:04.989412 (+ 286us) server_negotiation.cc:207] Beginning negotiation
1023 20:14:04.989428 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1023 20:14:04.989448 (+ 20us) server_negotiation.cc:408] Connection header received
1023 20:14:04.989868 (+ 420us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:14:04.989874 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:14:04.989929 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:14:04.990057 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:14:04.990981 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:14:04.992335 (+ 1354us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:14:04.993183 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:14:04.993445 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:14:04.996259 (+ 2814us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:14:04.996267 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:14:04.996267 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:14:04.996292 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:14:05.001977 (+ 5685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:14:05.002454 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:14:05.002458 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:14:05.002458 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:14:05.002530 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:14:05.002743 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:14:05.002744 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:14:05.002744 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:14:05.003053 (+ 309us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:14:05.003184 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1023 20:14:05.003733 (+ 549us) server_negotiation.cc:300] Negotiation successful
1023 20:14:05.003864 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":200,"threads_started":1}
I20251023 20:14:05.007015 21668 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:14:05.007607 21668 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:14:05.014422 21668 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1ba41f83d6e240b0bdd7979fe7afae71: Generated new TSK 0
I20251023 20:14:05.014767 21668 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:14:06.044128 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:33047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:34405
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:14:06.140041 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:14:06.140247 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:14:06.140286 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:14:06.142470 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:14:06.142534 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:14:06.142582 21695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:14:06.144816 21695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.117.212:33047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:34405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:14:06.145301 21695 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:14:06.145560 21695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:14:06.148809 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:14:06.154413 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:14:06.154994 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:14:06.155566 21695 server_base.cc:1047] running on GCE node
I20251023 20:14:06.156104 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:14:06.156410 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:14:06.159727 21695 hybrid_clock.cc:648] HybridClock initialized: now 1761250446159687 us; error 39 us; skew 500 ppm
Oct 23 20:14:06 dist-test-slave-x62p krb5kdc[21110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:14:06.163210 21695 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:14:06.163867 21695 webserver.cc:492] Webserver started at http://127.9.117.193:37421/ using document root <none> and password file <none>
I20251023 20:14:06.164088 21695 fs_manager.cc:362] Metadata directory not provided
I20251023 20:14:06.164132 21695 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:14:06.164260 21695 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 23 20:14:06 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:14:06.181941 21695 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "d38beb7f803e4d6093553604d1af13b0"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "b26988e23aa1e83d768898e9b1f7e8ff"
server_key_iv: "ae58edb837601c4756f2bc7b478f8027"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.182467 21695 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "d38beb7f803e4d6093553604d1af13b0"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "b26988e23aa1e83d768898e9b1f7e8ff"
server_key_iv: "ae58edb837601c4756f2bc7b478f8027"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.184998 21695 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20251023 20:14:06.204775 21710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.205041 21695 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20251023 20:14:06.205127 21695 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "d38beb7f803e4d6093553604d1af13b0"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "b26988e23aa1e83d768898e9b1f7e8ff"
server_key_iv: "ae58edb837601c4756f2bc7b478f8027"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.205235 21695 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:14:06.231112 21695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:14:06.232870 21695 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:14:06.232987 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:14:06.233148 21695 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:14:06.233462 21695 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:14:06.233493 21695 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.233518 21695 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:14:06.233544 21695 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.241032 21695 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:34939
I20251023 20:14:06.241080 21823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:34939 every 8 connection(s)
I20251023 20:14:06.241468 21695 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:14:06.242015 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 21695
I20251023 20:14:06.242121 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:14:06.242337 9687 external_mini_cluster.cc:1468] Setting key 9843a2c8108bc2175ca2b2c39bddc2d5
Oct 23 20:14:06 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:14:06.251662 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:06.242336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:43655 (local address 127.9.117.254:34405)
1023 20:14:06.242508 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1023 20:14:06.242510 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:14:06.243200 (+ 690us) server_negotiation.cc:408] Connection header received
1023 20:14:06.244576 (+ 1376us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:14:06.244578 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:14:06.244597 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:14:06.244654 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:14:06.245557 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:14:06.246094 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:14:06.246712 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:14:06.246889 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:14:06.248875 (+ 1986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:14:06.248880 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:14:06.248880 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:14:06.248888 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:14:06.250388 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:14:06.250798 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:14:06.250799 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:14:06.250799 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:14:06.250826 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:14:06.251037 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:14:06.251037 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:14:06.251038 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:14:06.251132 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:14:06.251196 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:14:06.251463 (+ 267us) server_negotiation.cc:300] Negotiation successful
1023 20:14:06.251560 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20251023 20:14:06.251822 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:06.242308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:43655)
1023 20:14:06.243068 (+ 760us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:06.243104 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1023 20:14:06.244379 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:14:06.244813 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:14:06.244818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:14:06.245071 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:14:06.245464 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:14:06.245467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:14:06.246188 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:14:06.246189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:14:06.246633 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:14:06.246636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:14:06.246799 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:14:06.247388 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:14:06.247409 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:14:06.248775 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:14:06.250475 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:14:06.250483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:14:06.250487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:14:06.250729 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:14:06.250881 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:14:06.250882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:14:06.250882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:14:06.250964 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:14:06.251190 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:14:06.251195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:14:06.251339 (+ 144us) client_negotiation.cc:769] Sending connection context
1023 20:14:06.251441 (+ 102us) client_negotiation.cc:240] Negotiation successful
1023 20:14:06.251580 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":74,"threads_started":1}
I20251023 20:14:06.252445 21824 heartbeater.cc:344] Connected to a master server at 127.9.117.254:34405
I20251023 20:14:06.252552 21824 heartbeater.cc:461] Registering TS with master...
I20251023 20:14:06.252821 21824 heartbeater.cc:507] Master 127.9.117.254:34405 requested a full tablet report, sending...
I20251023 20:14:06.253368 21613 ts_manager.cc:194] Registered new tserver with Master: d38beb7f803e4d6093553604d1af13b0 (127.9.117.193:34939)
I20251023 20:14:06.254062 21613 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:43655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:14:06.295895 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:33047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:34405
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:14:06.389178 21832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:14:06.389463 21832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:14:06.389508 21832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:14:06.392168 21832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:14:06.392253 21832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:14:06.392300 21832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:14:06.395440 21832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.117.212:33047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:34405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:14:06.395790 21832 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:14:06.396231 21832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:14:06.399823 21842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:14:06.400844 21840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:14:06.401196 21839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:14:06.401520 21832 server_base.cc:1047] running on GCE node
I20251023 20:14:06.401784 21832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:14:06.402061 21832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:14:06.403242 21832 hybrid_clock.cc:648] HybridClock initialized: now 1761250446403222 us; error 37 us; skew 500 ppm
Oct 23 20:14:06 dist-test-slave-x62p krb5kdc[21110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:14:06.407240 21832 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:14:06.408043 21832 webserver.cc:492] Webserver started at http://127.9.117.194:35155/ using document root <none> and password file <none>
I20251023 20:14:06.408304 21832 fs_manager.cc:362] Metadata directory not provided
I20251023 20:14:06.408352 21832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:14:06.408504 21832 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 23 20:14:06 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:14:06.424371 21832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "66bcd4e07b024f59ac38ad1c5bcf4069"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "2e373f1a71b9720ad917e1d9103ddb64"
server_key_iv: "112296fb22d21146f6de74b87375195f"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.425006 21832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "66bcd4e07b024f59ac38ad1c5bcf4069"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "2e373f1a71b9720ad917e1d9103ddb64"
server_key_iv: "112296fb22d21146f6de74b87375195f"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.429973 21832 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20251023 20:14:06.446890 21849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.447124 21832 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20251023 20:14:06.447212 21832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "66bcd4e07b024f59ac38ad1c5bcf4069"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "2e373f1a71b9720ad917e1d9103ddb64"
server_key_iv: "112296fb22d21146f6de74b87375195f"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.447310 21832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:14:06.460872 21832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:14:06.463361 21832 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:14:06.463526 21832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:14:06.463769 21832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:14:06.464187 21832 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:14:06.464234 21832 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.464274 21832 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:14:06.464301 21832 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.472667 21832 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:38411
I20251023 20:14:06.473290 21832 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:14:06.475930 21964 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:38411 every 8 connection(s)
I20251023 20:14:06.482386 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 21832
I20251023 20:14:06.482514 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:14:06.482726 9687 external_mini_cluster.cc:1468] Setting key 041d15305b935820f33dcbf33a17f14e
Oct 23 20:14:06 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:14:06.491338 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:06.476934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:50183 (local address 127.9.117.254:34405)
1023 20:14:06.476992 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1023 20:14:06.476994 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:14:06.477334 (+ 340us) server_negotiation.cc:408] Connection header received
1023 20:14:06.478704 (+ 1370us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:14:06.478706 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:14:06.478734 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:14:06.478799 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:14:06.479809 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:14:06.480641 (+ 832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:14:06.481612 (+ 971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:14:06.481818 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:14:06.487336 (+ 5518us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:14:06.487342 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:14:06.487343 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:14:06.487352 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:14:06.489311 (+ 1959us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:14:06.490134 (+ 823us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:14:06.490136 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:14:06.490137 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:14:06.490168 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:14:06.490591 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:14:06.490593 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:14:06.490593 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:14:06.490697 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:14:06.490773 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1023 20:14:06.491113 (+ 340us) server_negotiation.cc:300] Negotiation successful
1023 20:14:06.491226 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251023 20:14:06.491829 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:06.476623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:50183)
1023 20:14:06.477210 (+ 587us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:06.477244 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1023 20:14:06.478544 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:14:06.478874 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:14:06.478880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:14:06.479203 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:14:06.479652 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:14:06.479658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:14:06.480825 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:14:06.480826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:14:06.481492 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:14:06.481494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:14:06.481667 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:14:06.483076 (+ 1409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:14:06.483111 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:14:06.486276 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:14:06.489642 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:14:06.489652 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:14:06.489656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:14:06.489994 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:14:06.490427 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:14:06.490428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:14:06.490428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:14:06.490511 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:14:06.490775 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:14:06.490783 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:14:06.490935 (+ 152us) client_negotiation.cc:769] Sending connection context
1023 20:14:06.491476 (+ 541us) client_negotiation.cc:240] Negotiation successful
1023 20:14:06.491636 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":113,"threads_started":1}
I20251023 20:14:06.492401 21965 heartbeater.cc:344] Connected to a master server at 127.9.117.254:34405
I20251023 20:14:06.492516 21965 heartbeater.cc:461] Registering TS with master...
I20251023 20:14:06.492808 21965 heartbeater.cc:507] Master 127.9.117.254:34405 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
I20251023 20:14:06.493372 21613 ts_manager.cc:194] Registered new tserver with Master: 66bcd4e07b024f59ac38ad1c5bcf4069 (127.9.117.194:38411)
I20251023 20:14:06.493937 21613 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:50183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
I20251023 20:14:06.544332 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:33047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:34405
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.9.117.254 from keytab /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20251023 20:14:06.565735 21687 server.cc:273] Received an EOF from the subprocess
F20251023 20:14:06.566756 21674 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f7127973dcd google::LogMessage::Fail() at ??:0
@ 0x7f7127977b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f71279737cc google::LogMessage::Flush() at ??:0
@ 0x7f7127974f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7128d79313 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f7128a6b4e7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7128d9a6db start_thread at ??:0
@ 0x7f7126ec871f clone at ??:0
W20251023 20:14:06.637658 21972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:14:06.637861 21972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:14:06.637902 21972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:14:06.639434 21972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:14:06.639485 21972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:14:06.639528 21972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:14:06.641294 21972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.117.212:33047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:34405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:14:06.641551 21972 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:14:06.641750 21972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:14:06.644331 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:14:06.644331 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:14:06.644526 21972 server_base.cc:1047] running on GCE node
W20251023 20:14:06.644632 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:14:06.644833 21972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:14:06.645066 21972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:14:06.646198 21972 hybrid_clock.cc:648] HybridClock initialized: now 1761250446646186 us; error 31 us; skew 500 ppm
Oct 23 20:14:06 dist-test-slave-x62p krb5kdc[21110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:14:06.648859 21972 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:14:06.649451 21972 webserver.cc:492] Webserver started at http://127.9.117.195:41765/ using document root <none> and password file <none>
I20251023 20:14:06.649652 21972 fs_manager.cc:362] Metadata directory not provided
I20251023 20:14:06.649694 21972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:14:06.649830 21972 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 23 20:14:06 dist-test-slave-x62p krb5kdc[21110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:14:06.664151 21972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "7f8047071dc24dc0bfe552bc0ddf7e01"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "69944198c3857724743ce430a37d9c65"
server_key_iv: "3dfd0e5e07859f0d52557640545ed8bc"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.664594 21972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "7f8047071dc24dc0bfe552bc0ddf7e01"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "69944198c3857724743ce430a37d9c65"
server_key_iv: "3dfd0e5e07859f0d52557640545ed8bc"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.666371 21972 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20251023 20:14:06.680097 21996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.680356 21972 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:14:06.680437 21972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "7f8047071dc24dc0bfe552bc0ddf7e01"
format_stamp: "Formatted at 2025-10-23 20:14:06 on dist-test-slave-x62p"
server_key: "69944198c3857724743ce430a37d9c65"
server_key_iv: "3dfd0e5e07859f0d52557640545ed8bc"
server_key_version: "kuduclusterkey@0"
I20251023 20:14:06.680537 21972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:14:06.702083 21972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:14:06.703902 21972 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:14:06.704059 21972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:14:06.704294 21972 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:14:06.704667 21972 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:14:06.704710 21972 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.704771 21972 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:14:06.704802 21972 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:14:06.712667 21972 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:37735
I20251023 20:14:06.712761 22109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:37735 every 8 connection(s)
I20251023 20:14:06.713092 21972 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:14:06.722775 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 21972
I20251023 20:14:06.722929 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:14:06.723148 9687 external_mini_cluster.cc:1468] Setting key 43be6bb2e9af5d0e5e16ce1a8957b64f
W20251023 20:14:06.829769 21108 connection.cc:537] client connection to 127.9.117.254:34405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:14:06.830127 22112 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:14:06.713829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:43461)
1023 20:14:06.713978 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:06.714011 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:14:06.714944 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:14:06.829918 (+114974us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20251023 20:14:06.830384 22110 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:34405 (0 consecutive failures): Network error: Failed to ping master at 127.9.117.254:34405: Client connection negotiation failed: client connection to 127.9.117.254:34405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:14:06.830684 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:06.830592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:59381)
1023 20:14:06.830638 (+ 46us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:06.830668 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20251023 20:14:06.830920 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:06.830860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:58175)
1023 20:14:06.830890 (+ 30us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:06.830908 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20251023 20:14:06.831012 22110 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251023 20:14:07.254984 21824 heartbeater.cc:499] Master 127.9.117.254:34405 was elected leader, sending a full tablet report...
I20251023 20:14:07.255537 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:07.255261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:45335)
1023 20:14:07.255403 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:07.255467 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
W20251023 20:14:07.255788 21824 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:34405 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
I20251023 20:14:07.494849 21965 heartbeater.cc:499] Master 127.9.117.254:34405 was elected leader, sending a full tablet report...
I20251023 20:14:07.495455 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:07.495178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:58115)
1023 20:14:07.495325 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:07.495392 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20251023 20:14:07.495641 21965 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:34405 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
I20251023 20:14:07.831725 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:07.831436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:37937)
1023 20:14:07.831601 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:07.831649 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":69,"threads_started":1}
I20251023 20:14:08.256477 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:08.256237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:49689)
1023 20:14:08.256373 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:08.256416 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20251023 20:14:08.496472 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:08.496159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:34167)
1023 20:14:08.496345 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:08.496392 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20251023 20:14:08.832573 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:08.832301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:57661)
1023 20:14:08.832449 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:08.832498 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20251023 20:14:09.257328 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:09.257049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:32911)
1023 20:14:09.257200 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:09.257248 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20251023 20:14:09.497407 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:09.497125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:42725)
1023 20:14:09.497300 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:09.497343 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20251023 20:14:09.833587 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:09.833278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:44383)
1023 20:14:09.833476 (+ 198us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:09.833527 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20251023 20:14:10.258191 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:10.257922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:49341)
1023 20:14:10.258090 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:10.258128 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20251023 20:14:10.498343 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:10.498044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:49541)
1023 20:14:10.498221 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:10.498267 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20251023 20:14:10.834651 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:10.834349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:51963)
1023 20:14:10.834541 (+ 192us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:10.834591 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20251023 20:14:11.259123 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:11.258819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:38611)
1023 20:14:11.259011 (+ 192us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:11.259062 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20251023 20:14:11.499176 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:11.498904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:43625)
1023 20:14:11.499078 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:11.499119 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20251023 20:14:11.835700 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:11.835383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:39383)
1023 20:14:11.835550 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:11.835599 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20251023 20:14:12.259965 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:12.259662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:36405)
1023 20:14:12.259852 (+ 190us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:12.259902 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20251023 20:14:12.500092 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:12.499783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:60227)
1023 20:14:12.499962 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:12.500010 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":77,"threads_started":1}
I20251023 20:14:12.836601 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:12.836337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:43521)
1023 20:14:12.836487 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:12.836527 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20251023 20:14:13.260926 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:13.260563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:56117)
1023 20:14:13.260808 (+ 245us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:13.260858 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":75,"threads_started":1}
I20251023 20:14:13.500975 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:13.500648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:51359)
1023 20:14:13.500873 (+ 225us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:13.500920 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":127,"threads_started":1}
I20251023 20:14:13.837376 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:13.837131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:38289)
1023 20:14:13.837272 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:13.837318 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20251023 20:14:14.261685 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:14.261409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:48109)
1023 20:14:14.261579 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:14.261615 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20251023 20:14:14.501886 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:14.501589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:55369)
1023 20:14:14.501775 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:14.501827 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20251023 20:14:14.838404 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:14.838141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:46515)
1023 20:14:14.838292 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:14.838330 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20251023 20:14:15.262575 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:15.262275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:34267)
1023 20:14:15.262462 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:15.262509 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20251023 20:14:15.502616 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:15.502397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:36599)
1023 20:14:15.502523 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:15.502562 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20251023 20:14:15.839390 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:15.839106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:37289)
1023 20:14:15.839282 (+ 176us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:15.839331 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20251023 20:14:16.263521 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:16.263191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:54679)
1023 20:14:16.263398 (+ 207us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:16.263448 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20251023 20:14:16.503482 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:16.503227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:54033)
1023 20:14:16.503378 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:16.503424 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20251023 20:14:16.840319 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:16.840025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:42191)
1023 20:14:16.840216 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:16.840262 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20251023 20:14:17.264515 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:17.264185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:41373)
1023 20:14:17.264395 (+ 210us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:17.264446 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20251023 20:14:17.504472 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:17.504136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:41341)
1023 20:14:17.504349 (+ 213us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:17.504399 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20251023 20:14:17.841362 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:17.841086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:56435)
1023 20:14:17.841249 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:17.841296 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20251023 20:14:18.265427 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:18.265178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:36593)
1023 20:14:18.265319 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:18.265365 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20251023 20:14:18.505309 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:18.505046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:53991)
1023 20:14:18.505208 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:18.505255 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20251023 20:14:18.842250 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:18.842010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:57603)
1023 20:14:18.842148 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:18.842194 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20251023 20:14:19.266292 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:19.266034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:41737)
1023 20:14:19.266178 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:19.266228 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20251023 20:14:19.506124 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:19.505866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:43043)
1023 20:14:19.506032 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:19.506071 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20251023 20:14:19.843092 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:19.842838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:60643)
1023 20:14:19.842985 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:19.843034 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20251023 20:14:20.267408 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:20.267051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:54873)
1023 20:14:20.267292 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:20.267343 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":87,"threads_started":1}
I20251023 20:14:20.507339 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:20.506701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:37405)
1023 20:14:20.507157 (+ 456us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:20.507197 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":250,"threads_started":1}
I20251023 20:14:20.843973 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:20.843710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.195:42741)
1023 20:14:20.843866 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:20.843912 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
I20251023 20:14:21.268371 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:21.268124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.193:59475)
1023 20:14:21.268268 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:21.268313 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20251023 20:14:21.508438 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:14:21.508171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:34405 (local address 127.9.117.194:60009)
1023 20:14:21.508335 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1023 20:14:21.508384 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1052: Failure
Failed
Bad status: Timed out: Timed out waiting for 3 TS(s) to register with all masters
I20251023 20:14:21.727192 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 21695
I20251023 20:14:21.732645 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 21832
I20251023 20:14:21.737641 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 21972
I20251023 20:14:21.743932 9687 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251023 20:14:22.091032 9687 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251023 20:14:22.091177 9687 mini_ranger.cc:66] Stopping Ranger...
I20251023 20:14:22.437002 9687 mini_ranger.cc:68] Stopped Ranger
2025-10-23 20:14:22.437 UTC [21141] LOG: received smart shutdown request
2025-10-23 20:14:22.441 UTC [21141] LOG: background worker "logical replication launcher" (PID 21149) exited with exit code 1
2025-10-23 20:14:22.441 UTC [21144] LOG: shutting down
2025-10-23 20:14:22.445 UTC [21144] LOG: checkpoint starting: shutdown immediate
2025-10-23 20:14:23.903 UTC [21144] LOG: checkpoint complete: wrote 2418 buffers (14.8%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.044 s, sync=1.403 s, total=1.462 s; sync files=1113, longest=0.008 s, average=0.002 s; distance=12108 kB, estimate=12108 kB; lsn=0/204DCD0, redo lsn=0/204DCD0
2025-10-23 20:14:23.912 UTC [21141] LOG: database system is shut down
2025-10-23T20:14:23Z chronyd exiting
I20251023 20:14:23.932835 9687 test_util.cc:183] -----------------------------------------------
I20251023 20:14:23.932931 9687 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761250300132822-9687-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (71132 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:14:23 dist-test-slave-x62p krb5kdc[22167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:14:23 dist-test-slave-x62p krb5kdc[22167](info): set up 2 sockets
Oct 23 20:14:23 dist-test-slave-x62p krb5kdc[22167](info): commencing operation
krb5kdc: starting...
W20251023 20:14:25.952858 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:14:26 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:14:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:14:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.117.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.117.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/rangeradmin_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/rangeradmin_127.9.117.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.117.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.117.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/rangerlookup_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/rangerlookup_127.9.117.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.117.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/HTTP_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/HTTP_127.9.117.212@KRBTEST.COM.keytab.
I20251023 20:14:26.133371 9687 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskut_hJr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/postgres -l logfile start
2025-10-23 20:14:29.714 UTC [22198] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-23 20:14:29.714 UTC [22198] LOG: listening on IPv4 address "127.9.117.212", port 32969
2025-10-23 20:14:29.719 UTC [22198] LOG: listening on Unix socket "/tmp/.s.PGSQL.32969"
2025-10-23 20:14:29.726 UTC [22203] LOG: database system was shut down at 2025-10-23 20:14:27 UTC
2025-10-23 20:14:29.733 UTC [22198] LOG: database system is ready to accept connections
I20251023 20:14:31.652945 9687 mini_postgres.cc:96] Postgres bound to 32969
2025-10-23 20:14:31.660 UTC [22209] FATAL: database "slave" does not exist
127.9.117.212:32969 - accepting connections
I20251023 20:14:31.661260 9687 mini_ranger.cc:161] Starting Ranger...
I20251023 20:14:31.681562 9687 mini_ranger.cc:84] Created miniranger Postgres user
I20251023 20:14:31.763562 9687 mini_ranger.cc:87] Created ranger Postgres database
I20251023 20:14:31.763648 9687 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-admin
2025-10-23 20:14:32,084 [I] DB FLAVOR :POSTGRES
2025-10-23 20:14:32,085 [I] --------- Verifying Ranger DB connection ---------
2025-10-23 20:14:32,085 [I] Checking connection..
2025-10-23 20:14:32,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-23 20:14:32,453 [I] Checking connection passed.
2025-10-23 20:14:32,453 [I] --------- Verifying version history table ---------
2025-10-23 20:14:32,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-23 20:14:32,832 [I] Table x_db_version_h does not exist in database ranger
2025-10-23 20:14:32,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-23 20:14:33,200 [I] Table x_db_version_h does not exist in database ranger
2025-10-23 20:14:33,200 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-23 20:14:33,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-23 20:14:33.552 UTC [22276] WARNING: there is no transaction in progress
2025-10-23 20:14:33,561 [I] create_dbversion_catalog.sql file imported successfully
2025-10-23 20:14:33,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-23 20:14:33,940 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-23 20:14:33,940 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-23 20:14:33,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-23 20:14:34,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-23 20:14:34,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-x62p.c.gcp-upstream.internal','N') ;"
2025-10-23 20:14:35,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-23 20:14:35,396 [I] Table x_portal_user does not exist in database ranger
2025-10-23 20:14:35,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-23 20:14:35,781 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-23 20:14:35,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-23 20:14:36,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-23 20:14:36,530 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-23 20:14:36,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-23 20:14:37.584 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.599 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.614 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.629 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.644 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.660 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.723 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.731 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.745 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.758 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.771 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.785 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.797 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.805 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.812 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:37.819 UTC [22393] WARNING: there is no transaction in progress
2025-10-23 20:14:38,482 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-23 20:14:38,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-23 20:14:38,860 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-23 20:14:38,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-23 20:14:39,244 [I] Table x_portal_user already exists in database 'ranger'
2025-10-23 20:14:39,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-23 20:14:39,625 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-23 20:14:39,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-23 20:14:39,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-23 20:14:40,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-x62p.c.gcp-upstream.internal';"
2025-10-23 20:14:40,679 [I] CORE_DB_SCHEMA import status has been updated
2025-10-23 20:14:40,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-23 20:14:41,065 [I] DB_PATCHES have already been applied
I20251023 20:14:41.072247 9687 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-admin:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskut_hJr/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251023 20:14:41.072322 9687 mini_ranger.cc:193] Using host: 127.9.117.212
I20251023 20:14:41.075337 9687 mini_ranger.cc:227] Ranger admin URL: http://127.9.117.212:33181
Oct 23, 2025 8:14:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 23, 2025 8:14:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 23, 2025 8:14:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-admin/ews/webapp
Oct 23, 2025 8:14:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 23, 2025 8:14:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 23, 2025 8:14:41 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 23, 2025 8:14:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-admin/ews/webapp].
Oct 23, 2025 8:14:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 23, 2025 8:14:42 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-33181"]
Oct 23, 2025 8:14:42 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 23, 2025 8:14:42 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 23, 2025 8:14:42 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 23, 2025 8:14:42 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 23, 2025 8:14:42 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 23, 2025 8:14:43 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 23, 2025 8:14:43 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 23, 2025 8:14:43 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 23, 2025 8:14:43 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20251023 20:14:43.381279 9687 mini_ranger.cc:160] Time spent starting Ranger: real 11.720s user 0.000s sys 0.009s
Oct 23, 2025 8:15:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 23, 2025 8:15:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 23, 2025 8:15:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 23, 2025 8:15:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 23, 2025 8:15:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 23, 2025 8:15:07 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-33181"]
I20251023 20:15:08.329738 9687 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.117.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.117.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/rangerkms_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.117.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/rangerkms_127.9.117.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.117.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.117.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/HTTP_127.9.117.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.117.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/HTTP_127.9.117.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 23 20:15:08 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250508, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251023 20:15:08.443267 9687 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251023 20:15:08.455268 9687 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251023 20:15:08.571312 9687 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251023 20:15:08.571406 9687 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms
I20251023 20:15:08.571420 9687 mini_ranger_kms.cc:227] Using postgres at 127.9.117.212:32969
2025-10-23 20:15:08,778 [I] DB FLAVOR :POSTGRES
2025-10-23 20:15:08,778 [I] --------- Verifying Ranger DB connection ---------
2025-10-23 20:15:08,778 [I] Checking connection
2025-10-23 20:15:08,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-23 20:15:09,165 [I] connection success
2025-10-23 20:15:09,166 [I] --------- Verifying Ranger DB tables ---------
2025-10-23 20:15:09,166 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-23 20:15:09,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-23 20:15:09,550 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-23 20:15:09,550 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-23 20:15:09,550 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-23 20:15:09,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.117.212:32969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-23 20:15:09,938 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251023 20:15:10.594520 9687 mini_ranger_kms.cc:310] Created kms service
I20251023 20:15:10.882203 9687 mini_ranger_kms.cc:326] Created kudu user
I20251023 20:15:10.940384 9687 mini_ranger_kms.cc:343] Created rangerkms user
Oct 23 20:15:10 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250508, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:11.402828 9687 mini_ranger_kms.cc:384] Added ranger policy
I20251023 20:15:11.402971 9687 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms:/tmp/dist-test-taskut_hJr/build/release/bin/postgresql.jar:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskut_hJr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251023 20:15:11.402992 9687 mini_ranger_kms.cc:240] Using host: 127.9.117.212
I20251023 20:15:11.407148 9687 mini_ranger_kms.cc:276] Ranger KMS PID: 22585
I20251023 20:15:11.407241 9687 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.117.212:44819
Oct 23, 2025 8:15:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 23, 2025 8:15:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 23, 2025 8:15:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 23, 2025 8:15:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 23, 2025 8:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-kms/ews/webapp].
Oct 23, 2025 8:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 23, 2025 8:15:13 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-44819"]
Oct 23, 2025 8:15:13 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 23, 2025 8:15:13 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 23, 2025 8:15:13 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20251023 20:15:13.831735 9687 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.388s user 0.000s sys 0.186s
I20251023 20:15:13.831849 9687 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251023 20:15:13.831912 9687 mini_ranger_kms.cc:401] 127.9.117.212:44819/kms/v1/keys
Oct 23, 2025 8:15:14 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 23 20:15:14 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761250514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.117.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Oct 23 20:15:14 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.117.212@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
Oct 23 20:15:15 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.117.212@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
Oct 23 20:15:17 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:18.011631 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37495
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37495
--ranger_config_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:44819/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:18.102785 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:18.103013 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:18.103039 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:18.104719 22638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:18.104820 22638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:18.104833 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:18.104846 22638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:18.104859 22638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:18.106927 22638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.117.212:44819/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37495
--ranger_config_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:18.107192 22638 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:18.107393 22638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:18.110822 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:18.110829 22638 server_base.cc:1047] running on GCE node
W20251023 20:15:18.110900 22646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:18.110800 22643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:18.111354 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:18.111635 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:18.112835 22638 hybrid_clock.cc:648] HybridClock initialized: now 1761250518112814 us; error 51 us; skew 500 ppm
Oct 23 20:15:18 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:18.115764 22638 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:15:18.116550 22638 webserver.cc:492] Webserver started at http://127.9.117.254:40225/ using document root <none> and password file <none>
I20251023 20:15:18.116842 22638 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:18.116899 22638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:18.117038 22638 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 23 20:15:18 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:18.174578 22638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "dc9b565ed7184b378956583662042595"
format_stamp: "Formatted at 2025-10-23 20:15:18 on dist-test-slave-x62p"
server_key: "b1333c4b2aa229490ffffd4da9b59f54"
server_key_iv: "1c962d268b1052c956ea17149de1b896"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:18.175042 22638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "dc9b565ed7184b378956583662042595"
format_stamp: "Formatted at 2025-10-23 20:15:18 on dist-test-slave-x62p"
server_key: "b1333c4b2aa229490ffffd4da9b59f54"
server_key_iv: "1c962d268b1052c956ea17149de1b896"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:18.176819 22638 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:18.194310 22653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:18.194547 22638 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:18.194628 22638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "dc9b565ed7184b378956583662042595"
format_stamp: "Formatted at 2025-10-23 20:15:18 on dist-test-slave-x62p"
server_key: "b1333c4b2aa229490ffffd4da9b59f54"
server_key_iv: "1c962d268b1052c956ea17149de1b896"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:18.194737 22638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:18.206621 22638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:18.212996 22638 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:18.213142 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:18.218114 22638 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37495
I20251023 20:15:18.218376 22705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37495 every 8 connection(s)
I20251023 20:15:18.218560 22638 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:18.219256 22706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.221760 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Bootstrap starting.
I20251023 20:15:18.222455 22706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.222754 22706 log.cc:826] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:18.223582 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: No bootstrap required, opened a new log
I20251023 20:15:18.225054 22706 raft_consensus.cc:359] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } }
I20251023 20:15:18.225211 22706 raft_consensus.cc:385] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.225251 22706 raft_consensus.cc:740] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9b565ed7184b378956583662042595, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.225401 22706 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } }
I20251023 20:15:18.225488 22706 raft_consensus.cc:399] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:15:18.225536 22706 raft_consensus.cc:493] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:15:18.225586 22706 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:18.226271 22706 raft_consensus.cc:515] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } }
I20251023 20:15:18.226411 22706 leader_election.cc:304] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc9b565ed7184b378956583662042595; no voters:
I20251023 20:15:18.226584 22706 leader_election.cc:290] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:15:18.226668 22709 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:18.226876 22706 sys_catalog.cc:565] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:18.226915 22709 raft_consensus.cc:697] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 1 LEADER]: Becoming Leader. State: Replica: dc9b565ed7184b378956583662042595, State: Running, Role: LEADER
I20251023 20:15:18.227149 22709 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } }
I20251023 20:15:18.227698 22710 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc9b565ed7184b378956583662042595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } } }
I20251023 20:15:18.227779 22710 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:18.228225 22711 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9b565ed7184b378956583662042595. Latest consensus state: current_term: 1 leader_uuid: "dc9b565ed7184b378956583662042595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } } }
I20251023 20:15:18.228296 22711 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:18.228709 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 22638
I20251023 20:15:18.228863 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:15:18.229018 22706 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251023 20:15:18.229053 9687 external_mini_cluster.cc:1468] Setting key 9b1916610088036325d5d767839fb57e
I20251023 20:15:18.229074 22716 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:18.232417 22716 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 23 20:15:18 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:18.247805 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:18.232085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.9.117.254:37495)
1023 20:15:18.232527 (+ 442us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:18.232546 (+ 19us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:18.232580 (+ 34us) server_negotiation.cc:408] Connection header received
1023 20:15:18.232949 (+ 369us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:18.232956 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:18.233035 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:18.234301 (+ 1266us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:18.234734 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:18.236180 (+ 1446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:18.241108 (+ 4928us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:18.241499 (+ 391us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:18.241634 (+ 135us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:18.241646 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:18.241647 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:18.241667 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:18.245364 (+ 3697us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:18.245904 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:18.245918 (+ 14us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:18.245918 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:18.245984 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:18.246171 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:18.246172 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:18.246172 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:18.246494 (+ 322us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:18.247222 (+ 728us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:18.247313 (+ 91us) server_negotiation.cc:300] Negotiation successful
1023 20:15:18.247446 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":150,"threads_started":1}
I20251023 20:15:18.252882 22716 catalog_manager.cc:1357] Generated new cluster ID: 6e7d041919364366881bd15e0b4f9f7e
I20251023 20:15:18.252977 22716 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:15:18.280095 22716 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:15:18.281200 22716 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:15:18.291965 22716 catalog_manager.cc:6022] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Generated new TSK 0
I20251023 20:15:18.292363 22716 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:19.145133 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:44819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37495
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:19.258407 22753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:19.258664 22753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:19.258708 22753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:19.261037 22753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:19.261114 22753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:19.261170 22753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:15:19.263348 22753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.117.212:44819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:19.263667 22753 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:19.263912 22753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:19.266682 22758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:19.266682 22759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:19.266682 22761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:19.267568 22753 server_base.cc:1047] running on GCE node
I20251023 20:15:19.267808 22753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:19.268059 22753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:19.269198 22753 hybrid_clock.cc:648] HybridClock initialized: now 1761250519269184 us; error 32 us; skew 500 ppm
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:19.271976 22753 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:15:19.272593 22753 webserver.cc:492] Webserver started at http://127.9.117.193:35703/ using document root <none> and password file <none>
I20251023 20:15:19.272836 22753 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:19.272876 22753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:19.273001 22753 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:19.296029 22753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "d40a709cf6a049a985f4f8f1ef717d3f"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "2dc7c8842fae5bc1ec1f918f7249f1fa"
server_key_iv: "b341ca43069e04e2058a37c63f220292"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.296500 22753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "d40a709cf6a049a985f4f8f1ef717d3f"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "2dc7c8842fae5bc1ec1f918f7249f1fa"
server_key_iv: "b341ca43069e04e2058a37c63f220292"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.300102 22753 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20251023 20:15:19.324234 22770 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.324546 22753 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20251023 20:15:19.324690 22753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "d40a709cf6a049a985f4f8f1ef717d3f"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "2dc7c8842fae5bc1ec1f918f7249f1fa"
server_key_iv: "b341ca43069e04e2058a37c63f220292"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.325016 22753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:19.346845 22753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:19.349107 22753 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:19.349263 22753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:19.349458 22753 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:19.349823 22753 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:19.349864 22753 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.349898 22753 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:19.349952 22753 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.357427 22753 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:46411
I20251023 20:15:19.358202 22753 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:15:19.367131 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 22753
I20251023 20:15:19.367247 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:15:19.367453 9687 external_mini_cluster.cc:1468] Setting key 07ede2ae058471ebc635bba55863dbd0
I20251023 20:15:19.360787 22883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:46411 every 8 connection(s)
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:19.378997 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:19.361455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37495 (local address 127.9.117.193:36241)
1023 20:15:19.364886 (+ 3431us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:19.364935 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:19.366003 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:19.366386 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:19.366392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:19.366722 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:19.367237 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:19.367245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.369191 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:19.369192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:19.369867 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:19.369870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.370081 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:19.372950 (+ 2869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:19.373001 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:19.374750 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:19.377220 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:19.377231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:19.377235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:19.377604 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:19.377818 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:19.377819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:19.377820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:19.377912 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:19.378211 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:19.378219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:19.378417 (+ 198us) client_negotiation.cc:769] Sending connection context
1023 20:15:19.378548 (+ 131us) client_negotiation.cc:240] Negotiation successful
1023 20:15:19.378728 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3360,"thread_start_us":92,"threads_started":1}
I20251023 20:15:19.379683 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:19.361770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:36241 (local address 127.9.117.254:37495)
1023 20:15:19.362068 (+ 298us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:19.362071 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:19.365067 (+ 2996us) server_negotiation.cc:408] Connection header received
1023 20:15:19.366142 (+ 1075us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:19.366145 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:19.366167 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:19.366236 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:19.367399 (+ 1163us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.369053 (+ 1654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:19.370400 (+ 1347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.370616 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:19.374923 (+ 4307us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:19.374931 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:19.374932 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:19.374943 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:19.377089 (+ 2146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:19.377716 (+ 627us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:19.377718 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:19.377718 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:19.377753 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:19.377983 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:19.377984 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:19.377984 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:19.378115 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:19.378183 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:19.379441 (+ 1258us) server_negotiation.cc:300] Negotiation successful
1023 20:15:19.379570 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":60,"threads_started":1}
I20251023 20:15:19.380565 22884 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37495
I20251023 20:15:19.380687 22884 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:19.380981 22884 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:19.381579 22670 ts_manager.cc:194] Registered new tserver with Master: d40a709cf6a049a985f4f8f1ef717d3f (127.9.117.193:46411)
I20251023 20:15:19.382673 22670 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:36241
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:19.440371 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:44819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37495
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20251023 20:15:19.553357 22894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:19.553579 22894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:19.553622 22894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:19.555430 22894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:19.555500 22894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:19.555552 22894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:15:19.557750 22894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.117.212:44819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:19.558040 22894 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:19.558282 22894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:19.561496 22900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:19.561527 22902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:19.561861 22899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:19.562036 22894 server_base.cc:1047] running on GCE node
I20251023 20:15:19.562238 22894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:19.562506 22894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:19.563719 22894 hybrid_clock.cc:648] HybridClock initialized: now 1761250519563696 us; error 39 us; skew 500 ppm
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:19.567232 22894 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:15:19.567953 22894 webserver.cc:492] Webserver started at http://127.9.117.194:44189/ using document root <none> and password file <none>
I20251023 20:15:19.568204 22894 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:19.568252 22894 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:19.568395 22894 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:19.590092 22894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "5c7ec3272f7e493e8500e64c1a187026"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "d5b62cbc86343aa3c114fa3719f8a320"
server_key_iv: "cf7dd59ceb3de23503dd5f3a8115bb43"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.590713 22894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "5c7ec3272f7e493e8500e64c1a187026"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "d5b62cbc86343aa3c114fa3719f8a320"
server_key_iv: "cf7dd59ceb3de23503dd5f3a8115bb43"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.594480 22894 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:19.610102 22910 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.610399 22894 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:19.610479 22894 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "5c7ec3272f7e493e8500e64c1a187026"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "d5b62cbc86343aa3c114fa3719f8a320"
server_key_iv: "cf7dd59ceb3de23503dd5f3a8115bb43"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.610572 22894 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:19.627493 22894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:19.632057 22894 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:19.632233 22894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:19.632475 22894 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:19.632936 22894 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:19.632980 22894 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.633015 22894 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:19.633047 22894 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.641007 22894 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:40279
I20251023 20:15:19.641068 23023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:40279 every 8 connection(s)
I20251023 20:15:19.641450 22894 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:19.648768 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 22894
I20251023 20:15:19.648901 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:15:19.649116 9687 external_mini_cluster.cc:1468] Setting key ff9c0696ac1e1089eb3ed01d33d2890a
I20251023 20:15:19.651511 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:19.642272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:39265 (local address 127.9.117.254:37495)
1023 20:15:19.642349 (+ 77us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:19.642353 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:19.642511 (+ 158us) server_negotiation.cc:408] Connection header received
1023 20:15:19.643656 (+ 1145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:19.643658 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:19.643677 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:19.643727 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:19.644681 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.645283 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:19.645951 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.646122 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:19.648162 (+ 2040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:19.648168 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:19.648169 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:19.648179 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:19.650034 (+ 1855us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:19.650484 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:19.650485 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:19.650485 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:19.650516 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:19.650750 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:19.650756 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:19.650757 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:19.650862 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:19.650928 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:19.651260 (+ 332us) server_negotiation.cc:300] Negotiation successful
1023 20:15:19.651406 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:15:19.651690 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:19.642241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37495 (local address 127.9.117.194:39265)
1023 20:15:19.642395 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:19.642424 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:19.643522 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:19.643858 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:19.643863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:19.644192 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:19.644565 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:19.644570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.645385 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:19.645386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:19.645846 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:19.645849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.645992 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:19.646609 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:19.646626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:19.648057 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:19.650134 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:19.650144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:19.650150 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:19.650405 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:19.650576 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:19.650577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:19.650577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:19.650689 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:19.650924 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:19.650931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:19.651090 (+ 159us) client_negotiation.cc:769] Sending connection context
1023 20:15:19.651209 (+ 119us) client_negotiation.cc:240] Negotiation successful
1023 20:15:19.651397 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20251023 20:15:19.652684 23024 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37495
I20251023 20:15:19.652838 23024 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:19.653116 23024 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:19.653549 22670 ts_manager.cc:194] Registered new tserver with Master: 5c7ec3272f7e493e8500e64c1a187026 (127.9.117.194:40279)
I20251023 20:15:19.654139 22670 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:39265
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:19.700192 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:44819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37495
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:19.799996 23031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:19.800256 23031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:19.800302 23031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:19.802357 23031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:19.802426 23031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:19.802485 23031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:15:19.804584 23031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.117.212:44819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:19.804930 23031 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:19.805184 23031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:19.808382 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:19.808616 23037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:19.809123 23039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:19.809203 23031 server_base.cc:1047] running on GCE node
I20251023 20:15:19.809442 23031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:19.809731 23031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:19.810922 23031 hybrid_clock.cc:648] HybridClock initialized: now 1761250519810885 us; error 48 us; skew 500 ppm
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:19.815101 23031 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:15:19.815979 23031 webserver.cc:492] Webserver started at http://127.9.117.195:39533/ using document root <none> and password file <none>
I20251023 20:15:19.816256 23031 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:19.816308 23031 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:19.816438 23031 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:19.832507 23031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "6ea2ec09140ecf5bc595f666eb85c54b"
server_key_iv: "6c77618ae247fcba2e86b31b7f864a8a"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.833043 23031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "6ea2ec09140ecf5bc595f666eb85c54b"
server_key_iv: "6c77618ae247fcba2e86b31b7f864a8a"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.834952 23031 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:19.851513 23046 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.851850 23031 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:19.851967 23031 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "6ea2ec09140ecf5bc595f666eb85c54b"
server_key_iv: "6c77618ae247fcba2e86b31b7f864a8a"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:19.852095 23031 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:19.876595 23031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:19.879030 23031 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:19.879259 23031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:19.879566 23031 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:19.879995 23031 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:19.880034 23031 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.880070 23031 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:19.880092 23031 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:19.888180 23031 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:35171
I20251023 20:15:19.888635 23031 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:15:19.890208 23159 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:35171 every 8 connection(s)
Oct 23 20:15:19 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250519, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:19.898900 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 23031
I20251023 20:15:19.899006 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:15:19.899190 9687 external_mini_cluster.cc:1468] Setting key 4488c6233e24e571efbfdc4cc1afef61
I20251023 20:15:19.903944 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:19.891254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:33773 (local address 127.9.117.254:37495)
1023 20:15:19.891317 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:19.891322 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:19.891553 (+ 231us) server_negotiation.cc:408] Connection header received
1023 20:15:19.892709 (+ 1156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:19.892711 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:19.892756 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:19.892815 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:19.894126 (+ 1311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.894996 (+ 870us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:19.895952 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.896145 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:19.900235 (+ 4090us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:19.900240 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:19.900241 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:19.900250 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:19.902395 (+ 2145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:19.902962 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:19.902965 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:19.902965 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:19.903000 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:19.903194 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:19.903195 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:19.903195 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:19.903347 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:19.903425 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:19.903713 (+ 288us) server_negotiation.cc:300] Negotiation successful
1023 20:15:19.903836 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:15:19.903995 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:19.891217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37495 (local address 127.9.117.195:33773)
1023 20:15:19.891407 (+ 190us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:19.891438 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:19.892574 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:19.892905 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:19.892910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:19.893236 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:19.893957 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:19.893964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.895141 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:19.895143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:19.895813 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:19.895816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:19.896316 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:19.897090 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:19.897136 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:19.900102 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:19.902541 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:19.902553 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:19.902557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:19.902866 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:19.903070 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:19.903070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:19.903071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:19.903140 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:19.903407 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:19.903410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:19.903555 (+ 145us) client_negotiation.cc:769] Sending connection context
1023 20:15:19.903670 (+ 115us) client_negotiation.cc:240] Negotiation successful
1023 20:15:19.903799 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20251023 20:15:19.904517 23160 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37495
I20251023 20:15:19.904633 23160 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:19.904968 23160 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:19.905514 22670 ts_manager.cc:194] Registered new tserver with Master: fcb4cec6bb47416e9c9b7f6034fb4afb (127.9.117.195:35171)
I20251023 20:15:19.906113 22670 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:33773
I20251023 20:15:19.911754 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:15:19.912513 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 22753
I20251023 20:15:19.919636 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 22894
I20251023 20:15:19.925597 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 23031
I20251023 20:15:19.931772 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 22638
I20251023 20:15:19.936813 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37495
--webserver_interface=127.9.117.254
--webserver_port=40225
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37495
--ranger_config_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:44819/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.117.212:44819/kms with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:20.020514 23163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:20.020745 23163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:20.020773 23163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:20.022315 23163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:20.022367 23163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:20.022382 23163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:20.022393 23163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:20.022405 23163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:20.024147 23163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.117.212:44819/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37495
--ranger_config_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=40225
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:20.024374 23163 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:20.024569 23163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:15:20.027253 23163 server_base.cc:1047] running on GCE node
W20251023 20:15:20.027187 23171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:20.027189 23168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:20.027200 23169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:20.027709 23163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:20.027976 23163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:20.029129 23163 hybrid_clock.cc:648] HybridClock initialized: now 1761250520029107 us; error 35 us; skew 500 ppm
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:20.031970 23163 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:15:20.032582 23163 webserver.cc:492] Webserver started at http://127.9.117.254:40225/ using document root <none> and password file <none>
I20251023 20:15:20.032821 23163 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:20.032887 23163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:20.072896 23163 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:20.073830 23179 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.074134 23163 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:20.074223 23163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "dc9b565ed7184b378956583662042595"
format_stamp: "Formatted at 2025-10-23 20:15:18 on dist-test-slave-x62p"
server_key: "b1333c4b2aa229490ffffd4da9b59f54"
server_key_iv: "1c962d268b1052c956ea17149de1b896"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:20.074573 23163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:20.106300 23163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:20.108412 23163 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:20.108558 23163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:20.112991 23163 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37495
I20251023 20:15:20.113217 23231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37495 every 8 connection(s)
I20251023 20:15:20.113423 23163 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:20.114684 23232 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:15:20.115365 23232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Bootstrap starting.
I20251023 20:15:20.123009 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 23163
I20251023 20:15:20.123413 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:44819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:46411
--local_ip_for_outbound_sockets=127.9.117.193
--tserver_master_addrs=127.9.117.254:37495
--webserver_port=35703
--webserver_interface=127.9.117.193
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.117.212:44819/kms with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:15:20.125747 23232 log.cc:826] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:20.127660 23232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:20.127959 23232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Bootstrap complete.
I20251023 20:15:20.130116 23232 raft_consensus.cc:359] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } }
I20251023 20:15:20.130339 23232 raft_consensus.cc:740] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9b565ed7184b378956583662042595, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.130486 23232 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } }
I20251023 20:15:20.130568 23232 raft_consensus.cc:399] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:15:20.130618 23232 raft_consensus.cc:493] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:15:20.130672 23232 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:20.131664 23232 raft_consensus.cc:515] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } }
I20251023 20:15:20.131778 23232 leader_election.cc:304] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc9b565ed7184b378956583662042595; no voters:
I20251023 20:15:20.131966 23232 leader_election.cc:290] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251023 20:15:20.132022 23236 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:15:20.132176 23236 raft_consensus.cc:697] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [term 2 LEADER]: Becoming Leader. State: Replica: dc9b565ed7184b378956583662042595, State: Running, Role: LEADER
I20251023 20:15:20.132249 23232 sys_catalog.cc:565] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:20.132385 23236 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } }
I20251023 20:15:20.132834 23237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc9b565ed7184b378956583662042595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } } }
I20251023 20:15:20.132869 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9b565ed7184b378956583662042595. Latest consensus state: current_term: 2 leader_uuid: "dc9b565ed7184b378956583662042595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b565ed7184b378956583662042595" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37495 } } }
I20251023 20:15:20.132932 23237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:20.132957 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:20.133198 23242 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:20.133829 23242 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:20.134047 23242 catalog_manager.cc:1269] Loaded cluster ID: 6e7d041919364366881bd15e0b4f9f7e
I20251023 20:15:20.134135 23242 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:15:20.134327 23232 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251023 20:15:20.138583 23242 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:15:20.139235 23242 catalog_manager.cc:6033] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Loaded TSK: 0
I20251023 20:15:20.139592 23242 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251023 20:15:20.246032 23234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:20.246377 23234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:20.246441 23234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:20.249403 23234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:20.249501 23234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:20.249559 23234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:15:20.253082 23234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.117.212:44819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:46411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=35703
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:20.253427 23234 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:20.253731 23234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:20.257615 23270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:20.257735 23269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:20.257616 23272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:20.257961 23234 server_base.cc:1047] running on GCE node
I20251023 20:15:20.258209 23234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:20.258481 23234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:20.259649 23234 hybrid_clock.cc:648] HybridClock initialized: now 1761250520259632 us; error 31 us; skew 500 ppm
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:20.264967 23234 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:15:20.265892 23234 webserver.cc:492] Webserver started at http://127.9.117.193:35703/ using document root <none> and password file <none>
I20251023 20:15:20.266232 23234 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:20.266340 23234 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:20.294546 23234 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:20.295559 23280 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.295755 23234 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:20.295838 23234 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "d40a709cf6a049a985f4f8f1ef717d3f"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "2dc7c8842fae5bc1ec1f918f7249f1fa"
server_key_iv: "b341ca43069e04e2058a37c63f220292"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:20.296250 23234 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:20.317224 23234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:20.319978 23234 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:20.320204 23234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:20.320516 23234 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:20.321005 23234 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:20.321111 23234 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.321221 23234 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:20.321302 23234 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.328573 23234 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:46411
I20251023 20:15:20.329731 23234 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:15:20.332322 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 23234
I20251023 20:15:20.332701 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:44819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:40279
--local_ip_for_outbound_sockets=127.9.117.194
--tserver_master_addrs=127.9.117.254:37495
--webserver_port=44189
--webserver_interface=127.9.117.194
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.117.212:44819/kms with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:20.347533 23393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:46411 every 8 connection(s)
I20251023 20:15:20.360579 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:20.330825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37495 (local address 127.9.117.193:52901)
1023 20:15:20.331255 (+ 430us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:20.331294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:20.332047 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:20.344985 (+ 12938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:20.344992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:20.346144 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:20.346664 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:20.346673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.348637 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:20.348639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:20.349349 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:20.349352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.349501 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:20.350168 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:20.350212 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:20.352010 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:20.358798 (+ 6788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:20.358811 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:20.358814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:20.359159 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:20.359417 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:20.359418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:20.359418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:20.359497 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:20.359895 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:20.359900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:20.360058 (+ 158us) client_negotiation.cc:769] Sending connection context
1023 20:15:20.360144 (+ 86us) client_negotiation.cc:240] Negotiation successful
1023 20:15:20.360315 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":91,"threads_started":1}
I20251023 20:15:20.361770 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:20.331205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:52901 (local address 127.9.117.254:37495)
1023 20:15:20.340865 (+ 9660us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:20.340885 (+ 20us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:20.340940 (+ 55us) server_negotiation.cc:408] Connection header received
1023 20:15:20.341129 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:20.341137 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:20.341206 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:20.346871 (+ 5665us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:20.347048 (+ 177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.348464 (+ 1416us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:20.349833 (+ 1369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.350962 (+ 1129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:20.352168 (+ 1206us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:20.352199 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:20.352200 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:20.352231 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:20.358645 (+ 6414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:20.359275 (+ 630us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:20.359278 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:20.359278 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:20.359344 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:20.359564 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:20.359565 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:20.359565 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:20.359817 (+ 252us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:20.361337 (+ 1520us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:20.361431 (+ 94us) server_negotiation.cc:300] Negotiation successful
1023 20:15:20.361568 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9591,"thread_start_us":88,"threads_started":1}
I20251023 20:15:20.362813 23394 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37495
I20251023 20:15:20.362951 23394 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.363178 23394 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
W20251023 20:15:20.364110 23394 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:37495 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20251023 20:15:20.364198 23394 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.364326 23394 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:20.364886 23394 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.365015 23394 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
W20251023 20:15:20.365439 23394 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251023 20:15:20.456262 23398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:20.456588 23398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:20.456669 23398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:20.459529 23398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:20.459669 23398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:20.459754 23398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:15:20.462070 23398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.117.212:44819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:40279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=44189
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:20.462522 23398 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:20.462769 23398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:20.466085 23403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:20.466297 23406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:20.466406 23404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:20.466677 23398 server_base.cc:1047] running on GCE node
I20251023 20:15:20.466900 23398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:20.467139 23398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:20.468322 23398 hybrid_clock.cc:648] HybridClock initialized: now 1761250520468296 us; error 41 us; skew 500 ppm
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:20.473331 23398 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:15:20.474156 23398 webserver.cc:492] Webserver started at http://127.9.117.194:44189/ using document root <none> and password file <none>
I20251023 20:15:20.474404 23398 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:20.474455 23398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:20.494010 23398 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:20.499622 23414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.500937 23398 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.000s sys 0.001s
I20251023 20:15:20.501078 23398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "5c7ec3272f7e493e8500e64c1a187026"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "d5b62cbc86343aa3c114fa3719f8a320"
server_key_iv: "cf7dd59ceb3de23503dd5f3a8115bb43"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:20.501485 23398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:20.512701 23398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:20.514921 23398 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:20.515072 23398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:20.515275 23398 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:20.515654 23398 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:20.515698 23398 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.515733 23398 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:20.515786 23398 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.523975 23398 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:40279
I20251023 20:15:20.524459 23398 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:15:20.526290 23527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:40279 every 8 connection(s)
I20251023 20:15:20.530849 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 23398
I20251023 20:15:20.531214 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.117.212:44819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:35171
--local_ip_for_outbound_sockets=127.9.117.195
--tserver_master_addrs=127.9.117.254:37495
--webserver_port=39533
--webserver_interface=127.9.117.195
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.117.212:44819/kms with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:20.536971 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:20.526615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37495 (local address 127.9.117.194:40231)
1023 20:15:20.526801 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:20.526833 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:20.527863 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:20.528148 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:20.528153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:20.528398 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:20.528779 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:20.528784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.529522 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:20.529523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:20.529943 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:20.529945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.530131 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:20.530761 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:20.530798 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:20.532566 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:20.534940 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:20.534951 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:20.534954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:20.535327 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:20.535546 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:20.535547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:20.535548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:20.535665 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:20.536027 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:20.536034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:20.536253 (+ 219us) client_negotiation.cc:769] Sending connection context
1023 20:15:20.536395 (+ 142us) client_negotiation.cc:240] Negotiation successful
1023 20:15:20.536582 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":126,"threads_started":1}
I20251023 20:15:20.537098 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:20.526604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:40231 (local address 127.9.117.254:37495)
1023 20:15:20.526682 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:20.526685 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:20.526926 (+ 241us) server_negotiation.cc:408] Connection header received
1023 20:15:20.527999 (+ 1073us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:20.528001 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:20.528027 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:20.528078 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:20.528877 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.529419 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:20.530031 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.530247 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:20.532704 (+ 2457us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:20.532712 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:20.532712 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:20.532752 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:20.534835 (+ 2083us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:20.535431 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:20.535433 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:20.535433 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:20.535471 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:20.535757 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:20.535758 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:20.535759 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:20.535923 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:20.535997 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:20.536853 (+ 856us) server_negotiation.cc:300] Negotiation successful
1023 20:15:20.536990 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251023 20:15:20.537793 23528 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37495
I20251023 20:15:20.537925 23528 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.538213 23528 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
W20251023 20:15:20.538983 23528 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:37495 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20251023 20:15:20.539064 23528 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.539189 23528 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:20.539575 23528 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.539708 23528 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
W20251023 20:15:20.540082 23528 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251023 20:15:20.642953 23531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:20.643172 23531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:20.643217 23531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:20.644845 23531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:20.644902 23531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:20.644948 23531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:15:20.647038 23531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.117.212:44819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:35171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=39533
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:20.647321 23531 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:20.647593 23531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:20.654309 23536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:20.654309 23539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:20.654641 23537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:20.655632 23531 server_base.cc:1047] running on GCE node
I20251023 20:15:20.655864 23531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:20.656158 23531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:20.657310 23531 hybrid_clock.cc:648] HybridClock initialized: now 1761250520657288 us; error 35 us; skew 500 ppm
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:20.660254 23531 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:15:20.660995 23531 webserver.cc:492] Webserver started at http://127.9.117.195:39533/ using document root <none> and password file <none>
I20251023 20:15:20.661217 23531 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:20.661309 23531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:20.683782 23531 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:20.684621 23547 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.684881 23531 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:20.684988 23531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb"
format_stamp: "Formatted at 2025-10-23 20:15:19 on dist-test-slave-x62p"
server_key: "6ea2ec09140ecf5bc595f666eb85c54b"
server_key_iv: "6c77618ae247fcba2e86b31b7f864a8a"
server_key_version: "kuduclusterkey@0"
I20251023 20:15:20.685412 23531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:20.734396 23531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:20.737231 23531 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:20.737468 23531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:20.737774 23531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:20.738252 23531 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:20.738308 23531 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.738344 23531 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:20.738361 23531 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:20.747931 23531 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:35171
I20251023 20:15:20.747969 23660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:35171 every 8 connection(s)
I20251023 20:15:20.748425 23531 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:15:20.755345 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 23531
Oct 23 20:15:20 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:20.760165 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:20.749318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:47829 (local address 127.9.117.254:37495)
1023 20:15:20.749374 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:20.749377 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:20.749557 (+ 180us) server_negotiation.cc:408] Connection header received
1023 20:15:20.750536 (+ 979us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:20.750539 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:20.750556 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:20.750649 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:20.751406 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.752224 (+ 818us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:20.752913 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.753108 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:20.756836 (+ 3728us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:20.756841 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:20.756841 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:20.756849 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:20.758676 (+ 1827us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:20.759106 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:20.759107 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:20.759108 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:20.759137 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:20.759333 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:20.759334 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:20.759335 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:20.759449 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:20.759514 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:20.759932 (+ 418us) server_negotiation.cc:300] Negotiation successful
1023 20:15:20.760055 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:15:20.760347 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:20.749279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37495 (local address 127.9.117.195:47829)
1023 20:15:20.749437 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:20.749469 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:20.750411 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:20.750711 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:20.750715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:20.750965 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:20.751293 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:20.751296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.752335 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:20.752336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:20.752818 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:20.752822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.752959 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:20.753531 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:20.753556 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:20.756706 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:20.758754 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:20.758768 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:20.758771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:20.759033 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:20.759194 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:20.759195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:20.759196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:20.759277 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:20.759543 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:20.759551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:20.759749 (+ 198us) client_negotiation.cc:769] Sending connection context
1023 20:15:20.759886 (+ 137us) client_negotiation.cc:240] Negotiation successful
1023 20:15:20.760078 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20251023 20:15:20.761056 23661 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37495
I20251023 20:15:20.761207 23661 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.761482 23661 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
W20251023 20:15:20.762300 23661 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:37495 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20251023 20:15:20.762395 23661 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.762511 23661 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:20.762925 23661 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.763049 23661 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
W20251023 20:15:20.763736 23661 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251023 20:15:20.765218 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:20.755830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.9.117.254:37495)
1023 20:15:20.756396 (+ 566us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:20.756398 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:20.756405 (+ 7us) server_negotiation.cc:408] Connection header received
1023 20:15:20.756428 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:20.756428 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:20.756440 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:20.756505 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:20.757740 (+ 1235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.758514 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:20.759417 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:20.759608 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:20.762289 (+ 2681us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:20.762293 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:20.762294 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:20.762302 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:20.764129 (+ 1827us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:20.764448 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:20.764449 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:20.764449 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:20.764474 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:20.764612 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:20.764613 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:20.764614 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:20.764760 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:20.764954 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:20.764993 (+ 39us) server_negotiation.cc:300] Negotiation successful
1023 20:15:20.765099 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":86,"threads_started":1}
I20251023 20:15:21.365626 23394 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:21.365865 23394 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:21.366356 23196 ts_manager.cc:194] Registered new tserver with Master: d40a709cf6a049a985f4f8f1ef717d3f (127.9.117.193:46411)
I20251023 20:15:21.367453 23196 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:52901
Oct 23 20:15:21 dist-test-slave-x62p krb5kdc[22167](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761250521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:21.540270 23528 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:21.540509 23528 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:21.540915 23196 ts_manager.cc:194] Registered new tserver with Master: 5c7ec3272f7e493e8500e64c1a187026 (127.9.117.194:40279)
I20251023 20:15:21.541524 23196 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:40231
Oct 23 20:15:21 dist-test-slave-x62p krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for HTTP/127.9.117.212@KRBTEST.COM
I20251023 20:15:21.763957 23661 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:21.764214 23661 heartbeater.cc:507] Master 127.9.117.254:37495 requested a full tablet report, sending...
I20251023 20:15:21.764645 23196 ts_manager.cc:194] Registered new tserver with Master: fcb4cec6bb47416e9c9b7f6034fb4afb (127.9.117.195:35171)
I20251023 20:15:21.765280 23196 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:47829
I20251023 20:15:21.765740 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:15:21.774093 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.766816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.9.117.254:37495)
1023 20:15:21.767083 (+ 267us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:21.767086 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:21.767100 (+ 14us) server_negotiation.cc:408] Connection header received
1023 20:15:21.767128 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:21.767129 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:21.767142 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:21.767205 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:21.768122 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.768991 (+ 869us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.769718 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.769912 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.771051 (+ 1139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:21.771057 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:21.771058 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:21.771067 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:21.772840 (+ 1773us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:21.773203 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:21.773205 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:21.773205 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:21.773232 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:21.773395 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:21.773396 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:21.773397 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:21.773506 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:21.773607 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:21.773749 (+ 142us) server_negotiation.cc:300] Negotiation successful
1023 20:15:21.773863 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":74,"threads_started":1}
I20251023 20:15:21.776176 23196 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.778009 23196 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.785120 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.781342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:46411 (local address 127.0.0.1:49492)
1023 20:15:21.781807 (+ 465us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:21.781816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:21.781866 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:21.782173 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:21.782174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:21.782177 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:21.782406 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.782407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.783594 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.783595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:21.784766 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.784767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.784850 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.784899 (+ 49us) client_negotiation.cc:769] Sending connection context
1023 20:15:21.784955 (+ 56us) client_negotiation.cc:240] Negotiation successful
1023 20:15:21.785012 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":64,"threads_started":1}
I20251023 20:15:21.785120 23692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.781168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:35171 (local address 127.0.0.1:33788)
1023 20:15:21.781697 (+ 529us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:21.781707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:21.781790 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:21.782285 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:21.782286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:21.782289 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:21.782473 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.782475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.783721 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.783722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:21.784763 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.784767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.784847 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.784899 (+ 52us) client_negotiation.cc:769] Sending connection context
1023 20:15:21.784955 (+ 56us) client_negotiation.cc:240] Negotiation successful
1023 20:15:21.785012 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":104,"threads_started":1}
I20251023 20:15:21.785974 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.781318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.9.117.195:35171)
1023 20:15:21.781634 (+ 316us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:21.781638 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:21.781761 (+ 123us) server_negotiation.cc:408] Connection header received
1023 20:15:21.782095 (+ 334us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:21.782097 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:21.782116 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:21.782165 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:21.782568 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.783621 (+ 1053us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.784866 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.785753 (+ 887us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.785817 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:21.785853 (+ 36us) server_negotiation.cc:300] Negotiation successful
1023 20:15:21.785888 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":50,"threads_started":1}
I20251023 20:15:21.786844 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.781468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.9.117.193:46411)
1023 20:15:21.781621 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:21.781626 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:21.781937 (+ 311us) server_negotiation.cc:408] Connection header received
1023 20:15:21.781986 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:21.781988 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:21.782009 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:21.782073 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:21.782489 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.783499 (+ 1010us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.786119 (+ 2620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.786643 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.786698 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:21.786730 (+ 32us) server_negotiation.cc:300] Negotiation successful
1023 20:15:21.786766 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20251023 20:15:21.787446 23328 tablet_service.cc:1505] Processing CreateTablet for tablet 96c7ea01aa4d48b3aac843069c2aba2f (DEFAULT_TABLE table=test-table [id=a0c137079b0845eaaa1e766592cbc476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.787761 23328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c7ea01aa4d48b3aac843069c2aba2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.790529 23694 tablet_bootstrap.cc:492] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f: Bootstrap starting.
I20251023 20:15:21.791225 23694 tablet_bootstrap.cc:654] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:21.791527 23694 log.cc:826] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:21.792307 23694 tablet_bootstrap.cc:492] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f: No bootstrap required, opened a new log
I20251023 20:15:21.792387 23694 ts_tablet_manager.cc:1403] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:21.793129 23595 tablet_service.cc:1505] Processing CreateTablet for tablet 96c7ea01aa4d48b3aac843069c2aba2f (DEFAULT_TABLE table=test-table [id=a0c137079b0845eaaa1e766592cbc476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.793444 23595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c7ea01aa4d48b3aac843069c2aba2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.794751 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.781081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40279 (local address 127.0.0.1:57652)
1023 20:15:21.781567 (+ 486us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:21.781599 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:21.781701 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:21.782095 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:21.782097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:21.782101 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:21.782371 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.782374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.793381 (+ 11007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.793383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:21.794473 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.794477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.794566 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.794571 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:15:21.794604 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:15:21.794654 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":95,"threads_started":1}
I20251023 20:15:21.794787 23694 raft_consensus.cc:359] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.794960 23694 raft_consensus.cc:385] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:21.794996 23694 raft_consensus.cc:740] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40a709cf6a049a985f4f8f1ef717d3f, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.795113 23694 consensus_queue.cc:260] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.795387 23694 ts_tablet_manager.cc:1434] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:15:21.795547 23394 heartbeater.cc:499] Master 127.9.117.254:37495 was elected leader, sending a full tablet report...
I20251023 20:15:21.795850 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.781189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.9.117.194:40279)
1023 20:15:21.781484 (+ 295us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:21.781489 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:21.781675 (+ 186us) server_negotiation.cc:408] Connection header received
1023 20:15:21.781858 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:21.781861 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:21.781885 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:21.781977 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:21.782462 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.793234 (+ 10772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.794986 (+ 1752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.795605 (+ 619us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.795673 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:21.795714 (+ 41us) server_negotiation.cc:300] Negotiation successful
1023 20:15:21.795760 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":75,"threads_started":1}
I20251023 20:15:21.796455 23697 tablet_bootstrap.cc:492] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: Bootstrap starting.
I20251023 20:15:21.796479 23462 tablet_service.cc:1505] Processing CreateTablet for tablet 96c7ea01aa4d48b3aac843069c2aba2f (DEFAULT_TABLE table=test-table [id=a0c137079b0845eaaa1e766592cbc476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.796900 23462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c7ea01aa4d48b3aac843069c2aba2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.797294 23697 tablet_bootstrap.cc:654] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:21.797606 23697 log.cc:826] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:21.798498 23697 tablet_bootstrap.cc:492] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: No bootstrap required, opened a new log
I20251023 20:15:21.798563 23697 ts_tablet_manager.cc:1403] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:21.799556 23699 tablet_bootstrap.cc:492] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: Bootstrap starting.
I20251023 20:15:21.800145 23699 tablet_bootstrap.cc:654] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:21.800446 23699 log.cc:826] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:21.800570 23697 raft_consensus.cc:359] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.800767 23697 raft_consensus.cc:385] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:21.800809 23697 raft_consensus.cc:740] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb4cec6bb47416e9c9b7f6034fb4afb, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.800917 23697 consensus_queue.cc:260] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.801216 23697 ts_tablet_manager.cc:1434] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251023 20:15:21.801241 23661 heartbeater.cc:499] Master 127.9.117.254:37495 was elected leader, sending a full tablet report...
I20251023 20:15:21.801543 23699 tablet_bootstrap.cc:492] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: No bootstrap required, opened a new log
I20251023 20:15:21.801622 23699 ts_tablet_manager.cc:1403] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:21.803372 23699 raft_consensus.cc:359] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.803498 23699 raft_consensus.cc:385] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:21.803519 23699 raft_consensus.cc:740] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7ec3272f7e493e8500e64c1a187026, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.803601 23699 consensus_queue.cc:260] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.803854 23699 ts_tablet_manager.cc:1434] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:21.803941 23528 heartbeater.cc:499] Master 127.9.117.254:37495 was elected leader, sending a full tablet report...
W20251023 20:15:21.857065 23395 tablet.cc:2378] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:21.900368 23696 raft_consensus.cc:493] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.900528 23696 raft_consensus.cc:515] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.900992 23696 leader_election.cc:290] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c7ec3272f7e493e8500e64c1a187026 (127.9.117.194:40279), fcb4cec6bb47416e9c9b7f6034fb4afb (127.9.117.195:35171)
I20251023 20:15:21.905342 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.901462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:55847 (local address 127.9.117.194:40279)
1023 20:15:21.901518 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:21.901521 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:21.901740 (+ 219us) server_negotiation.cc:408] Connection header received
1023 20:15:21.901839 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:21.901841 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:21.901860 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:21.901895 (+ 35us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:21.902427 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.903236 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.904535 (+ 1299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.905060 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.905073 (+ 13us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:21.905211 (+ 138us) server_negotiation.cc:300] Negotiation successful
1023 20:15:21.905257 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:15:21.905342 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.901474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:57925 (local address 127.9.117.195:35171)
1023 20:15:21.901562 (+ 88us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:21.901564 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:21.901847 (+ 283us) server_negotiation.cc:408] Connection header received
1023 20:15:21.901968 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:21.901970 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:21.901985 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:21.902017 (+ 32us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:21.902479 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.903234 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.904534 (+ 1300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.905065 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.905073 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:21.905212 (+ 139us) server_negotiation.cc:300] Negotiation successful
1023 20:15:21.905257 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251023 20:15:21.905658 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.901198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:35171 (local address 127.9.117.193:57925)
1023 20:15:21.901778 (+ 580us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:21.901788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:21.901874 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:21.902112 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:21.902114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:21.902115 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:21.902389 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.902391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.903353 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.903354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:21.904415 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.904419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.905110 (+ 691us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.905114 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:15:21.905493 (+ 379us) client_negotiation.cc:240] Negotiation successful
1023 20:15:21.905562 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":33,"thread_start_us":96,"threads_started":1}
I20251023 20:15:21.906062 23601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96c7ea01aa4d48b3aac843069c2aba2f" candidate_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" is_pre_election: true
I20251023 20:15:21.906100 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.901198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:40279 (local address 127.9.117.193:55847)
1023 20:15:21.901642 (+ 444us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:21.901654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:21.901733 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:21.901992 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:21.901993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:21.901995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:21.902332 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.902334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.903351 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.903352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:21.904435 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:21.904436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.905110 (+ 674us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.905114 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:15:21.905906 (+ 792us) client_negotiation.cc:240] Negotiation successful
1023 20:15:21.906014 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":133,"threads_started":1}
I20251023 20:15:21.906241 23601 raft_consensus.cc:2468] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40a709cf6a049a985f4f8f1ef717d3f in term 0.
I20251023 20:15:21.906484 23283 leader_election.cc:304] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d40a709cf6a049a985f4f8f1ef717d3f, fcb4cec6bb47416e9c9b7f6034fb4afb; no voters:
I20251023 20:15:21.906493 23469 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96c7ea01aa4d48b3aac843069c2aba2f" candidate_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7ec3272f7e493e8500e64c1a187026" is_pre_election: true
I20251023 20:15:21.906630 23696 raft_consensus.cc:2804] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:21.906667 23469 raft_consensus.cc:2468] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40a709cf6a049a985f4f8f1ef717d3f in term 0.
I20251023 20:15:21.906703 23696 raft_consensus.cc:493] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:21.906754 23696 raft_consensus.cc:3060] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:21.907684 23696 raft_consensus.cc:515] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.907848 23696 leader_election.cc:290] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [CANDIDATE]: Term 1 election: Requested vote from peers 5c7ec3272f7e493e8500e64c1a187026 (127.9.117.194:40279), fcb4cec6bb47416e9c9b7f6034fb4afb (127.9.117.195:35171)
I20251023 20:15:21.908138 23469 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96c7ea01aa4d48b3aac843069c2aba2f" candidate_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7ec3272f7e493e8500e64c1a187026"
I20251023 20:15:21.908151 23601 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96c7ea01aa4d48b3aac843069c2aba2f" candidate_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb"
I20251023 20:15:21.908234 23469 raft_consensus.cc:3060] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:21.908267 23601 raft_consensus.cc:3060] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:21.909283 23469 raft_consensus.cc:2468] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40a709cf6a049a985f4f8f1ef717d3f in term 1.
I20251023 20:15:21.909361 23601 raft_consensus.cc:2468] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40a709cf6a049a985f4f8f1ef717d3f in term 1.
I20251023 20:15:21.909559 23283 leader_election.cc:304] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d40a709cf6a049a985f4f8f1ef717d3f, fcb4cec6bb47416e9c9b7f6034fb4afb; no voters:
I20251023 20:15:21.909647 23696 raft_consensus.cc:2804] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:21.909804 23696 raft_consensus.cc:697] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [term 1 LEADER]: Becoming Leader. State: Replica: d40a709cf6a049a985f4f8f1ef717d3f, State: Running, Role: LEADER
I20251023 20:15:21.909925 23696 consensus_queue.cc:237] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } }
I20251023 20:15:21.910765 23195 catalog_manager.cc:5649] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d40a709cf6a049a985f4f8f1ef717d3f (127.9.117.193). New cstate: current_term: 1 leader_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40a709cf6a049a985f4f8f1ef717d3f" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:21.931264 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.928743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.9.117.193:46411)
1023 20:15:21.928803 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:21.928805 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:21.928830 (+ 25us) server_negotiation.cc:408] Connection header received
1023 20:15:21.928876 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:21.928877 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:21.928896 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:21.928948 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:15:21.929304 (+ 356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.929840 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:21.930671 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:21.930821 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:21.930885 (+ 64us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:15:21.930996 (+ 111us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:15:21.931063 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:21.931152 (+ 89us) server_negotiation.cc:300] Negotiation successful
1023 20:15:21.931181 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251023 20:15:21.933427 23469 raft_consensus.cc:1275] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 1 FOLLOWER]: Refusing update from remote peer d40a709cf6a049a985f4f8f1ef717d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:21.933426 23601 raft_consensus.cc:1275] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 1 FOLLOWER]: Refusing update from remote peer d40a709cf6a049a985f4f8f1ef717d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:21.933897 23696 consensus_queue.cc:1048] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7ec3272f7e493e8500e64c1a187026" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 40279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:21.934103 23705 consensus_queue.cc:1048] T 96c7ea01aa4d48b3aac843069c2aba2f P d40a709cf6a049a985f4f8f1ef717d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb4cec6bb47416e9c9b7f6034fb4afb" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:21.938060 23710 mvcc.cc:204] Tried to move back new op lower bound from 7214082137835982848 to 7214082137743646720. Current Snapshot: MvccSnapshot[applied={T|T < 7214082137835982848}]
I20251023 20:15:21.938936 23708 mvcc.cc:204] Tried to move back new op lower bound from 7214082137835982848 to 7214082137743646720. Current Snapshot: MvccSnapshot[applied={T|T < 7214082137835982848}]
I20251023 20:15:21.939127 23709 mvcc.cc:204] Tried to move back new op lower bound from 7214082137835982848 to 7214082137743646720. Current Snapshot: MvccSnapshot[applied={T|T < 7214082137835982848}]
I20251023 20:15:21.942054 23195 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48806:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:15:21.942173 23195 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48806:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:15:21.943017 23195 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc9b565ed7184b378956583662042595: Sending DeleteTablet for 3 replicas of tablet 96c7ea01aa4d48b3aac843069c2aba2f
I20251023 20:15:21.943624 23462 tablet_service.cc:1552] Processing DeleteTablet for tablet 96c7ea01aa4d48b3aac843069c2aba2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:57652
I20251023 20:15:21.943900 23328 tablet_service.cc:1552] Processing DeleteTablet for tablet 96c7ea01aa4d48b3aac843069c2aba2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:49492
I20251023 20:15:21.944128 23595 tablet_service.cc:1552] Processing DeleteTablet for tablet 96c7ea01aa4d48b3aac843069c2aba2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:33788
I20251023 20:15:21.944768 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 23234
I20251023 20:15:21.946025 23718 tablet_replica.cc:333] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: stopping tablet replica
I20251023 20:15:21.946149 23718 raft_consensus.cc:2243] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:21.946282 23718 raft_consensus.cc:2272] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:21.946874 23718 ts_tablet_manager.cc:1916] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:21.948973 23718 ts_tablet_manager.cc:1929] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:21.949121 23718 log.cc:1199] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/96c7ea01aa4d48b3aac843069c2aba2f
I20251023 20:15:21.949551 23718 ts_tablet_manager.cc:1950] T 96c7ea01aa4d48b3aac843069c2aba2f P 5c7ec3272f7e493e8500e64c1a187026: Deleting consensus metadata
I20251023 20:15:21.950556 23183 catalog_manager.cc:4985] TS 5c7ec3272f7e493e8500e64c1a187026 (127.9.117.194:40279): tablet 96c7ea01aa4d48b3aac843069c2aba2f (table test-table [id=a0c137079b0845eaaa1e766592cbc476]) successfully deleted
I20251023 20:15:21.950626 23720 tablet_replica.cc:333] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: stopping tablet replica
I20251023 20:15:21.950762 23720 raft_consensus.cc:2243] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:21.950909 23720 raft_consensus.cc:2272] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:21.951675 23183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:46411 (error 108)
I20251023 20:15:21.952000 23720 ts_tablet_manager.cc:1916] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:21.952354 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:21.952237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:46411 (local address 127.0.0.1:49508)
1023 20:15:21.952286 (+ 49us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:21.952315 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:46411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20251023 20:15:21.952525 23183 catalog_manager.cc:4712] TS d40a709cf6a049a985f4f8f1ef717d3f (127.9.117.193:46411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96c7ea01aa4d48b3aac843069c2aba2f: Network error: Client connection negotiation failed: client connection to 127.9.117.193:46411: connect: Connection refused (error 111)
I20251023 20:15:21.952913 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 23398
I20251023 20:15:21.954398 23720 ts_tablet_manager.cc:1929] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:21.954478 23720 log.cc:1199] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/96c7ea01aa4d48b3aac843069c2aba2f
I20251023 20:15:21.954810 23720 ts_tablet_manager.cc:1950] T 96c7ea01aa4d48b3aac843069c2aba2f P fcb4cec6bb47416e9c9b7f6034fb4afb: Deleting consensus metadata
I20251023 20:15:21.955287 23182 catalog_manager.cc:4985] TS fcb4cec6bb47416e9c9b7f6034fb4afb (127.9.117.195:35171): tablet 96c7ea01aa4d48b3aac843069c2aba2f (table test-table [id=a0c137079b0845eaaa1e766592cbc476]) successfully deleted
I20251023 20:15:21.959913 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 23531
I20251023 20:15:21.965091 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 23163
I20251023 20:15:21.971484 9687 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251023 20:15:22.316963 9687 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251023 20:15:22.317075 9687 mini_ranger.cc:66] Stopping Ranger...
I20251023 20:15:22.662747 9687 mini_ranger.cc:68] Stopped Ranger
2025-10-23 20:15:22.662 UTC [22198] LOG: received smart shutdown request
2025-10-23 20:15:22.666 UTC [22198] LOG: background worker "logical replication launcher" (PID 22206) exited with exit code 1
2025-10-23 20:15:22.667 UTC [22201] LOG: shutting down
2025-10-23 20:15:22.669 UTC [22201] LOG: checkpoint starting: shutdown immediate
2025-10-23 20:15:23.761 UTC [22201] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.042 s, sync=1.044 s, total=1.095 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12073 kB, estimate=12073 kB; lsn=0/2044D68, redo lsn=0/2044D68
2025-10-23 20:15:23.770 UTC [22198] LOG: database system is shut down
2025-10-23T20:15:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59980 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:15:23 dist-test-slave-x62p krb5kdc[23730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:15:23 dist-test-slave-x62p krb5kdc[23730](info): set up 2 sockets
Oct 23 20:15:23 dist-test-slave-x62p krb5kdc[23730](info): commencing operation
krb5kdc: starting...
W20251023 20:15:25.917259 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:15:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:26.072381 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33327
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33327,127.9.117.253:39265,127.9.117.252:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:26.160079 23746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:26.160297 23746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:26.160326 23746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:26.162091 23746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:26.162164 23746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:26.162180 23746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:26.162194 23746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:26.162205 23746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:26.164139 23746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33327,127.9.117.253:39265,127.9.117.252:42197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:26.164414 23746 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:26.164590 23746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:26.168023 23752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:26.168081 23751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:26.168105 23746 server_base.cc:1047] running on GCE node
W20251023 20:15:26.168023 23754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:26.168567 23746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:26.168949 23746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:26.170161 23746 hybrid_clock.cc:648] HybridClock initialized: now 1761250526170142 us; error 43 us; skew 500 ppm
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:26.172988 23746 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:15:26.173627 23746 webserver.cc:492] Webserver started at http://127.9.117.254:45147/ using document root <none> and password file <none>
I20251023 20:15:26.173840 23746 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:26.173887 23746 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:26.174000 23746 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:26.175084 23746 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "9544b1f0e6444a98b98ada8dd3761e1c"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "9bffdd1283f5258c4049cb246b30ece2"
server_key_iv: "c38e12a5ad0c23a3bbb1fc09efc6e5c2"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.175419 23746 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "9544b1f0e6444a98b98ada8dd3761e1c"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "9bffdd1283f5258c4049cb246b30ece2"
server_key_iv: "c38e12a5ad0c23a3bbb1fc09efc6e5c2"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.176833 23746 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:15:26.177682 23761 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:26.177947 23746 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:26.178030 23746 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "9544b1f0e6444a98b98ada8dd3761e1c"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "9bffdd1283f5258c4049cb246b30ece2"
server_key_iv: "c38e12a5ad0c23a3bbb1fc09efc6e5c2"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.178102 23746 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:26.191351 23746 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:26.193758 23746 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:26.193920 23746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:26.198638 23746 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:33327
I20251023 20:15:26.198689 23813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:33327 every 8 connection(s)
I20251023 20:15:26.199048 23746 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:26.199985 23814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.202112 23814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:26.207772 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 23746
I20251023 20:15:26.207885 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:15:26.208067 9687 external_mini_cluster.cc:1468] Setting key b1d5f738a9df0fa66a63e10e411ac6c8
I20251023 20:15:26.212829 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.203025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.9.117.254:33327)
1023 20:15:26.203171 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.203181 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.203205 (+ 24us) server_negotiation.cc:408] Connection header received
1023 20:15:26.203647 (+ 442us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.203650 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.203670 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.203754 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.204529 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.205325 (+ 796us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.205956 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.206128 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.208414 (+ 2286us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.208425 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.208425 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.208433 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.210862 (+ 2429us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.211408 (+ 546us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.211411 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.211412 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.211470 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.211721 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.211722 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.211722 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.211991 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.212115 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.212365 (+ 250us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.212612 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20251023 20:15:26.212841 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.202837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33327 (local address 127.0.0.1:37494)
1023 20:15:26.203063 (+ 226us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.203096 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.203484 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.203771 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.203774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.204101 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.204369 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.204372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.205424 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.205426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.205841 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.205843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.205973 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.206561 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.206579 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.208302 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.210985 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.210994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.210998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.211290 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.211552 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.211553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.211554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.211648 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.212085 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.212091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.212190 (+ 99us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.212313 (+ 123us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.212511 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20251023 20:15:26.213605 23814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:26.214020 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.213910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39614)
1023 20:15:26.213949 (+ 39us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.213996 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20251023 20:15:26.214144 23763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
I20251023 20:15:26.214743 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.214626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39622)
1023 20:15:26.214695 (+ 69us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.214721 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20251023 20:15:26.214869 23814 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:39265: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.9.117.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:26.264109 23814 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } attempt: 1
I20251023 20:15:26.264767 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.264515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39626)
1023 20:15:26.264604 (+ 89us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.264684 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20251023 20:15:26.264940 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.253:39265
--webserver_interface=127.9.117.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33327,127.9.117.253:39265,127.9.117.252:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:15:26.265328 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.265208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39636)
1023 20:15:26.265259 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.265292 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251023 20:15:26.265450 23814 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:39265: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
I20251023 20:15:26.333642 23814 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } attempt: 2
I20251023 20:15:26.334331 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.334142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39646)
1023 20:15:26.334233 (+ 91us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.334286 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20251023 20:15:26.334813 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.334691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39648)
1023 20:15:26.334751 (+ 60us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.334784 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20251023 20:15:26.334985 23814 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:39265: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39265: connect: Connection refused (error 111)
W20251023 20:15:26.353822 23823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:26.354054 23823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:26.354099 23823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:26.355932 23823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:26.356022 23823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:26.356050 23823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:26.356069 23823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:26.356091 23823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:26.358224 23823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33327,127.9.117.253:39265,127.9.117.252:42197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.253:39265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.117.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:26.358592 23823 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:26.358814 23823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:26.362234 23831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:26.362301 23828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:26.362591 23823 server_base.cc:1047] running on GCE node
W20251023 20:15:26.362234 23829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:26.362936 23823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:26.363222 23823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:26.364403 23823 hybrid_clock.cc:648] HybridClock initialized: now 1761250526364345 us; error 73 us; skew 500 ppm
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:26.367524 23823 init.cc:377] Logged in from keytab as kudu/127.9.117.253@KRBTEST.COM (short username kudu)
I20251023 20:15:26.368275 23823 webserver.cc:492] Webserver started at http://127.9.117.253:46857/ using document root <none> and password file <none>
I20251023 20:15:26.368497 23823 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:26.368541 23823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:26.368882 23823 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:26.370021 23823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/data/instance:
uuid: "cae26edcaf494f5c9570b7de5aac2021"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "99a4c08d2671d657e29483b78d1213b2"
server_key_iv: "833eca3208a65ef36aacdecb25495e5e"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.370374 23823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/wal/instance:
uuid: "cae26edcaf494f5c9570b7de5aac2021"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "99a4c08d2671d657e29483b78d1213b2"
server_key_iv: "833eca3208a65ef36aacdecb25495e5e"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.371968 23823 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:15:26.372959 23838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:26.373251 23823 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:26.373347 23823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/wal
uuid: "cae26edcaf494f5c9570b7de5aac2021"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "99a4c08d2671d657e29483b78d1213b2"
server_key_iv: "833eca3208a65ef36aacdecb25495e5e"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.373430 23823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:26.390727 23823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:26.393445 23823 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:26.393613 23823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:26.399305 23823 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.253:39265
I20251023 20:15:26.399386 23890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.253:39265 every 8 connection(s)
I20251023 20:15:26.399763 23823 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
I20251023 20:15:26.400575 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 23823
I20251023 20:15:26.400583 23891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.400707 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-1/wal/instance
I20251023 20:15:26.400928 9687 external_mini_cluster.cc:1468] Setting key b38eeaa70c5bfc7dc8bea99da7383998
I20251023 20:15:26.402758 23891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.117.252@KRBTEST.COM; defaulting to no policy
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.252@KRBTEST.COM" created.
I20251023 20:15:26.414208 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.403590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.9.117.254:33327)
1023 20:15:26.403678 (+ 88us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.403681 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.403957 (+ 276us) server_negotiation.cc:408] Connection header received
1023 20:15:26.404671 (+ 714us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.404673 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.404690 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.404810 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.406126 (+ 1316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.406652 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.407608 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.407781 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.410408 (+ 2627us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.410415 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.410416 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.410426 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.412370 (+ 1944us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.413152 (+ 782us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.413154 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.413154 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.413196 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.413443 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.413444 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.413444 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.413562 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.413630 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.413958 (+ 328us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.414087 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251023 20:15:26.414335 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.403556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33327 (local address 127.0.0.1:37498)
1023 20:15:26.403763 (+ 207us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.403812 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.404503 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.404902 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.404909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.405451 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.405976 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.405983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.406771 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.406773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.407446 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.407450 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.407635 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.408346 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.408395 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.410264 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.412512 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.412525 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.412528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.413053 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.413261 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.413262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.413262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.413368 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.413628 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.413637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.413809 (+ 172us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.413925 (+ 116us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.414106 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20251023 20:15:26.415475 23891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
I20251023 20:15:26.424925 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.415879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39658)
1023 20:15:26.415963 (+ 84us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.415976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.416031 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.416505 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.416506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.416720 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.416989 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.416993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.417983 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.417985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.418632 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.418635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.418731 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.419349 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.419356 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.421016 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.423797 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.423798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.423798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.423955 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.424149 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.424150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.424150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.424176 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.424429 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.424429 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.424471 (+ 42us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.424540 (+ 69us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.424676 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20251023 20:15:26.425048 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.416120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.9.117.253:39265)
1023 20:15:26.416350 (+ 230us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.416357 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.416375 (+ 18us) server_negotiation.cc:408] Connection header received
1023 20:15:26.416400 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.416403 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.416416 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.416481 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.417090 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.417871 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.418755 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.418991 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.421136 (+ 2145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.421151 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.421152 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.421159 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.423670 (+ 2511us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.424042 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.424046 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.424046 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.424090 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.424227 (+ 137us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.424228 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.424228 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.424369 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.424465 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.424580 (+ 115us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.424905 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":198,"thread_start_us":116,"threads_started":1}
I20251023 20:15:26.425946 23891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:26.426277 23814 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } attempt: 3
I20251023 20:15:26.426447 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.426251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43812)
1023 20:15:26.426339 (+ 88us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.426404 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20251023 20:15:26.426585 23842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
I20251023 20:15:26.427093 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.426987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43824)
1023 20:15:26.427029 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.427063 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20251023 20:15:26.427407 23891 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:42197: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:26.436165 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.426662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.9.117.253:39265)
1023 20:15:26.426707 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.426710 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.426720 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:15:26.426899 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.426900 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.426920 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.426980 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.427691 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.428412 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.429114 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.429287 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.431071 (+ 1784us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.431077 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.431078 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.431088 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.434950 (+ 3862us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.435392 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.435395 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.435396 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.435438 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.435630 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.435631 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.435631 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.435749 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.435806 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.435988 (+ 182us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.436086 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:15:26.436270 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.426595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39660)
1023 20:15:26.426659 (+ 64us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.426670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.426720 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.427007 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.427010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.427334 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.427566 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.427570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.428519 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.428521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.428989 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.428993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.429087 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.429608 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.429612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.430951 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.435111 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.435114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.435114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.435300 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.435521 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.435522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.435522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.435566 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.435808 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.435808 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.435877 (+ 69us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.435956 (+ 79us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.436112 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251023 20:15:26.437055 23814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:26.437565 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.437392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43840)
1023 20:15:26.437468 (+ 76us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.437516 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20251023 20:15:26.438064 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.437929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43850)
1023 20:15:26.437987 (+ 58us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.438025 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20251023 20:15:26.438186 23814 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:42197: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.9.117.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:26.466059 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.252:42197
--webserver_interface=127.9.117.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33327,127.9.117.253:39265,127.9.117.252:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:15:26.469336 23814 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } attempt: 1
I20251023 20:15:26.469966 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.469792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43852)
1023 20:15:26.469870 (+ 78us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.469912 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20251023 20:15:26.470431 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.470285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43868)
1023 20:15:26.470355 (+ 70us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.470399 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20251023 20:15:26.470552 23814 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:42197: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
I20251023 20:15:26.476704 23891 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } attempt: 1
I20251023 20:15:26.477484 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.477271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43882)
1023 20:15:26.477380 (+ 109us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.477447 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20251023 20:15:26.477938 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.477833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43892)
1023 20:15:26.477877 (+ 44us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.477905 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20251023 20:15:26.478041 23891 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:42197: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
I20251023 20:15:26.545742 23814 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } attempt: 2
I20251023 20:15:26.546205 23891 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } attempt: 2
I20251023 20:15:26.546350 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.546194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43906)
1023 20:15:26.546268 (+ 74us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.546303 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20251023 20:15:26.546669 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.546495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43918)
1023 20:15:26.546559 (+ 64us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.546599 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20251023 20:15:26.546774 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.546630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43926)
1023 20:15:26.546703 (+ 73us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.546741 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20251023 20:15:26.547061 23814 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:42197: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
I20251023 20:15:26.547163 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.547056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43936)
1023 20:15:26.547098 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.547128 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20251023 20:15:26.547349 23891 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:42197: Network error: Client connection negotiation failed: client connection to 127.9.117.252:42197: connect: Connection refused (error 111)
W20251023 20:15:26.555980 23900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:26.556242 23900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:26.556272 23900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:26.558207 23900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:26.558293 23900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:26.558310 23900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:26.558323 23900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:26.558336 23900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:26.560546 23900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33327,127.9.117.253:39265,127.9.117.252:42197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.252:42197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.117.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:26.560900 23900 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:26.561136 23900 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:26.564174 23906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:26.564232 23908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:26.564237 23900 server_base.cc:1047] running on GCE node
W20251023 20:15:26.564174 23905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:26.564721 23900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:26.565057 23900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:26.566195 23900 hybrid_clock.cc:648] HybridClock initialized: now 1761250526566180 us; error 35 us; skew 500 ppm
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:26.568924 23900 init.cc:377] Logged in from keytab as kudu/127.9.117.252@KRBTEST.COM (short username kudu)
I20251023 20:15:26.569568 23900 webserver.cc:492] Webserver started at http://127.9.117.252:45579/ using document root <none> and password file <none>
I20251023 20:15:26.569777 23900 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:26.569823 23900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:26.569962 23900 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:26.571134 23900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/data/instance:
uuid: "743e2243be854fb391ab804650234e3f"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "2178852f9f672a1557b9254393bcd22b"
server_key_iv: "fedd7c72c01df9947a2d4dc7d09d300b"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.571494 23900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/wal/instance:
uuid: "743e2243be854fb391ab804650234e3f"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "2178852f9f672a1557b9254393bcd22b"
server_key_iv: "fedd7c72c01df9947a2d4dc7d09d300b"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.573069 23900 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:15:26.574095 23915 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:26.574409 23900 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:26.574517 23900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/wal
uuid: "743e2243be854fb391ab804650234e3f"
format_stamp: "Formatted at 2025-10-23 20:15:26 on dist-test-slave-x62p"
server_key: "2178852f9f672a1557b9254393bcd22b"
server_key_iv: "fedd7c72c01df9947a2d4dc7d09d300b"
server_key_version: "encryptionkey@0"
I20251023 20:15:26.574621 23900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:26.585254 23900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:26.587677 23900 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:26.587844 23900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:26.592828 23900 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.252:42197
I20251023 20:15:26.592890 23967 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.252:42197 every 8 connection(s)
I20251023 20:15:26.593268 23900 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
I20251023 20:15:26.594015 23968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.596352 23968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:26.601692 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 23900
I20251023 20:15:26.601967 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761250300132822-9687-0/minicluster-data/master-2/wal/instance
I20251023 20:15:26.602197 9687 external_mini_cluster.cc:1468] Setting key 0b52af05b54d003f7d930f69b996f801
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:26.606770 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.597258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.9.117.254:33327)
1023 20:15:26.597337 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.597340 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.597560 (+ 220us) server_negotiation.cc:408] Connection header received
1023 20:15:26.598025 (+ 465us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.598027 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.598041 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.598106 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.599154 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.599630 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.600292 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.600437 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.603438 (+ 3001us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.603444 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.603445 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.603455 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.605130 (+ 1675us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.605776 (+ 646us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.605778 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.605778 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.605810 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.606046 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.606046 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.606047 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.606158 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.606226 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.606551 (+ 325us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.606666 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:15:26.606953 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.597218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:33327 (local address 127.0.0.1:37504)
1023 20:15:26.597421 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.597466 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.597890 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.598263 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.598267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.598628 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.598967 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.598970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.599746 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.599747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.600180 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.600182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.600311 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.601050 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.601081 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.603271 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.605229 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.605239 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.605243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.605668 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.605885 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.605886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.605886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.605979 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.606222 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.606229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.606403 (+ 174us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.606517 (+ 114us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.606697 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:15:26.607853 23968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:26.611943 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.602657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.9.117.252:42197)
1023 20:15:26.602861 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.602869 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.602894 (+ 25us) server_negotiation.cc:408] Connection header received
1023 20:15:26.602985 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.602987 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.603005 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.603073 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.603955 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.604687 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.605434 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.605655 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.608179 (+ 2524us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.608192 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.608193 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.608205 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.610395 (+ 2190us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.610790 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.610793 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.610793 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.610835 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.611275 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.611276 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.611277 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.611418 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.611655 (+ 237us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.611707 (+ 52us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.611807 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":121,"threads_started":1}
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
I20251023 20:15:26.615453 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.608201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39265 (local address 127.0.0.1:39670)
1023 20:15:26.608256 (+ 55us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.608267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.608302 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.608512 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.608513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.608676 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.608919 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.608923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.610271 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.610273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.610648 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.610649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.610725 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.611187 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.611192 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.612613 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.614507 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.614508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.614509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.614681 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.614843 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.614844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.614844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.614875 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.615113 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.615114 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.615161 (+ 47us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.615222 (+ 61us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.615344 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20251023 20:15:26.615453 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.608282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.9.117.253:39265)
1023 20:15:26.608342 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.608344 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.608353 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:15:26.608384 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.608385 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.608398 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.608476 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.609686 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.610177 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.610736 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.610930 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.612714 (+ 1784us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.612721 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.612721 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.612763 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.614414 (+ 1651us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.614753 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.614755 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.614755 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.614784 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.614928 (+ 144us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.614929 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.614929 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.615033 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.615100 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.615255 (+ 155us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.615356 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251023 20:15:26.616113 23968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:15:26.621219 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.613090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.9.117.254:33327)
1023 20:15:26.613149 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.613151 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.613160 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:15:26.613192 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.613193 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.613206 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.613267 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.614116 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.614574 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.615193 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.615693 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.618004 (+ 2311us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.618010 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.618011 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.618021 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.619659 (+ 1638us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.620096 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.620098 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.620098 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.620128 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.620383 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.620385 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.620385 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.620569 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.620795 (+ 226us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.621046 (+ 251us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.621140 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251023 20:15:26.623106 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.616418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43964)
1023 20:15:26.616464 (+ 46us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.616475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.616515 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.616673 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.616674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.616869 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.617072 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.617074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.617727 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.617729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.618204 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.618206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.618285 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.618757 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.618760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.620094 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.622149 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.622151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.622151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.622309 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.622490 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.622491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.622492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.622523 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.622746 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.622747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.622785 (+ 38us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.622853 (+ 68us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.622981 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20251023 20:15:26.623168 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.616483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.9.117.252:42197)
1023 20:15:26.616532 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.616535 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.616565 (+ 30us) server_negotiation.cc:408] Connection header received
1023 20:15:26.616591 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.616591 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.616605 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.616673 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.617152 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.617645 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.618309 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.618457 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.620187 (+ 1730us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.620192 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.620193 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.620201 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.622042 (+ 1841us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.622384 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.622385 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.622385 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.622414 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.622571 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.622572 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.622572 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.622686 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.622750 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.622886 (+ 136us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.623064 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":29}
I20251023 20:15:26.624526 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f: Bootstrap starting.
I20251023 20:15:26.625540 23968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.625876 23968 log.cc:826] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f: Log is configured to *not* fsync() on all Append() calls
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
I20251023 20:15:26.626739 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f: No bootstrap required, opened a new log
I20251023 20:15:26.628384 23968 raft_consensus.cc:359] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
I20251023 20:15:26.628527 23968 raft_consensus.cc:385] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.628566 23968 raft_consensus.cc:740] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743e2243be854fb391ab804650234e3f, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.628769 23968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
I20251023 20:15:26.629161 23968 sys_catalog.cc:565] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:26.629134 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.629285 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:26.630144 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.622490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.9.117.253:39265)
1023 20:15:26.622553 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.622555 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.622563 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:15:26.622591 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.622592 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.622601 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.622653 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.623352 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.623943 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.624493 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.624699 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.626939 (+ 2240us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.626945 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.626945 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.626953 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.628599 (+ 1646us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.629097 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.629099 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.629100 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.629137 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.629353 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.629355 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.629355 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.629510 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.629639 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.629878 (+ 239us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.630022 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20251023 20:15:26.631533 23985 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:15:26.631610 23985 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:15:26.632136 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:15:26.632071 23900 master_runner.cc:186] Error getting master registration for 127.9.117.254:33327: OK, instance_id { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" instance_seqno: 1761250526197233 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251023 20:15:26.633992 23900 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:15:26.638538 23891 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } attempt: 3
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:15:26.646209 23814 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } attempt: 3
I20251023 20:15:26.646546 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.638962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43976)
1023 20:15:26.639098 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.639110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.639169 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.639324 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.639325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.639502 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.639671 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.639674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.640564 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.640566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.641200 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.641205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.641310 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.641915 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.641922 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.643259 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.645482 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.645484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.645484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.645675 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.645853 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.645854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.645854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.645889 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.646152 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.646153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.646215 (+ 62us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.646283 (+ 68us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.646414 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20251023 20:15:26.646914 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.639043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.9.117.252:42197)
1023 20:15:26.639104 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.639107 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.639149 (+ 42us) server_negotiation.cc:408] Connection header received
1023 20:15:26.639251 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.639252 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.639261 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.639310 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.639754 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.640436 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.641312 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.641957 (+ 645us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.643372 (+ 1415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.643377 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.643377 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.643386 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.645353 (+ 1967us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.645766 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.645767 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.645767 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.645794 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.645945 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.645946 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.645946 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.646089 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.646156 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.646688 (+ 532us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.646801 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20251023 20:15:26.649456 23891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021: Bootstrap starting.
I20251023 20:15:26.650626 23891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.651204 23891 log.cc:826] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021: Log is configured to *not* fsync() on all Append() calls
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:15:26.654932 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.646812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.9.117.252:42197)
1023 20:15:26.647003 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.647005 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.647013 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:15:26.647039 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.647040 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.647054 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.647662 (+ 608us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.647705 (+ 43us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.648472 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.651986 (+ 3514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.652163 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.652205 (+ 42us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.652209 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.652209 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.652214 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.653854 (+ 1640us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.654226 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.654227 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.654228 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.654257 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.654403 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.654404 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.654405 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.654518 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.654584 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.654726 (+ 142us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.654823 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":128,"threads_started":1}
I20251023 20:15:26.654933 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.646551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:42197 (local address 127.0.0.1:43984)
1023 20:15:26.646617 (+ 66us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:26.646630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:26.646672 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:26.647175 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:26.647176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:26.647350 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:26.647552 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.647557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.648604 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.648606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:26.649206 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:26.649210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.649274 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.649920 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:26.649927 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:26.651851 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:26.653977 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.653979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.653979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.654150 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.654317 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:26.654318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:26.654319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:26.654350 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:26.654576 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:26.654578 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:26.654631 (+ 53us) client_negotiation.cc:769] Sending connection context
1023 20:15:26.654695 (+ 64us) client_negotiation.cc:240] Negotiation successful
1023 20:15:26.654827 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251023 20:15:26.656889 23814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c: Bootstrap starting.
I20251023 20:15:26.658150 23814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.658459 23814 log.cc:826] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:26.659399 23891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021: No bootstrap required, opened a new log
I20251023 20:15:26.661541 23891 raft_consensus.cc:359] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
I20251023 20:15:26.661872 23891 raft_consensus.cc:385] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.661975 23891 raft_consensus.cc:740] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae26edcaf494f5c9570b7de5aac2021, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.662187 23891 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
I20251023 20:15:26.662727 23891 sys_catalog.cc:565] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [sys.catalog]: configured and running, proceeding with master startup.
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:26.664183 23998 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.664283 23998 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [sys.catalog]: This master's current role is: FOLLOWER
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:15:26.665288 23823 master_runner.cc:186] Error getting master registration for 127.9.117.254:33327: OK, instance_id { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" instance_seqno: 1761250526197233 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251023 20:15:26.667583 23823 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:15:26.668532 23814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c: No bootstrap required, opened a new log
I20251023 20:15:26.669428 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.658139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.9.117.254:33327)
1023 20:15:26.658194 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.658197 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.658542 (+ 345us) server_negotiation.cc:408] Connection header received
1023 20:15:26.658602 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.658604 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.658618 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.658695 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.659751 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.660419 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.661329 (+ 910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.661515 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.665690 (+ 4175us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.665695 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.665696 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.665708 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.667987 (+ 2279us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.668517 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.668519 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.668519 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.668549 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.668779 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.668780 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.668780 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.668911 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.669004 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.669234 (+ 230us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.669339 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251023 20:15:26.669711 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.658766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.9.117.252:42197)
1023 20:15:26.658855 (+ 89us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.658857 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.658870 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:15:26.658903 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.658904 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.658914 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.658978 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.660106 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.661370 (+ 1264us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.662310 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.662509 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.665591 (+ 3082us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.665614 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.665615 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.665626 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.667917 (+ 2291us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.668507 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.668509 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.668509 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.668538 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.668698 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.668699 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.668699 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.668936 (+ 237us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.669050 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.669505 (+ 455us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.669596 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20251023 20:15:26.670222 24009 catalog_manager.cc:1568] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:15:26.670277 24009 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:15:26.670550 23814 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
I20251023 20:15:26.670656 23814 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.670689 23814 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9544b1f0e6444a98b98ada8dd3761e1c, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.670781 23814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
I20251023 20:15:26.671054 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.671124 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:26.671126 23814 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:26.672427 23974 raft_consensus.cc:493] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.672556 23974 raft_consensus.cc:515] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
I20251023 20:15:26.672863 23974 leader_election.cc:290] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9544b1f0e6444a98b98ada8dd3761e1c (127.9.117.254:33327), cae26edcaf494f5c9570b7de5aac2021 (127.9.117.253:39265)
I20251023 20:15:26.673488 23865 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "743e2243be854fb391ab804650234e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae26edcaf494f5c9570b7de5aac2021" is_pre_election: true
I20251023 20:15:26.673642 23865 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 743e2243be854fb391ab804650234e3f in term 0.
I20251023 20:15:26.673332 23788 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "743e2243be854fb391ab804650234e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" is_pre_election: true
I20251023 20:15:26.674016 23917 leader_election.cc:304] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 743e2243be854fb391ab804650234e3f, cae26edcaf494f5c9570b7de5aac2021; no voters:
I20251023 20:15:26.674054 23788 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 743e2243be854fb391ab804650234e3f in term 0.
I20251023 20:15:26.674158 23974 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.674213 23974 raft_consensus.cc:493] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.674238 23974 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.675279 23974 raft_consensus.cc:515] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
I20251023 20:15:26.675429 23974 leader_election.cc:290] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [CANDIDATE]: Term 1 election: Requested vote from peers 9544b1f0e6444a98b98ada8dd3761e1c (127.9.117.254:33327), cae26edcaf494f5c9570b7de5aac2021 (127.9.117.253:39265)
I20251023 20:15:26.676311 23788 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "743e2243be854fb391ab804650234e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9544b1f0e6444a98b98ada8dd3761e1c"
I20251023 20:15:26.676409 23788 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.677651 23788 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 743e2243be854fb391ab804650234e3f in term 1.
I20251023 20:15:26.678015 23916 leader_election.cc:304] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 743e2243be854fb391ab804650234e3f, 9544b1f0e6444a98b98ada8dd3761e1c; no voters:
I20251023 20:15:26.678177 23974 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.678336 23974 raft_consensus.cc:697] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [term 1 LEADER]: Becoming Leader. State: Replica: 743e2243be854fb391ab804650234e3f, State: Running, Role: LEADER
I20251023 20:15:26.678442 23974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } }
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
I20251023 20:15:26.678921 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743e2243be854fb391ab804650234e3f. Latest consensus state: current_term: 1 leader_uuid: "743e2243be854fb391ab804650234e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.679010 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:26.679515 23865 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "743e2243be854fb391ab804650234e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae26edcaf494f5c9570b7de5aac2021"
I20251023 20:15:26.679610 23865 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.680840 23865 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 743e2243be854fb391ab804650234e3f in term 1.
I20251023 20:15:26.681866 23746 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20251023 20:15:26.681958 23746 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:15:26.682862 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:26.673256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.9.117.253:39265)
1023 20:15:26.673309 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:26.673311 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:26.673320 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:15:26.673345 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:26.673346 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:26.673357 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:26.673426 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:26.674321 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.674979 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:26.676112 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:26.676335 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:26.679298 (+ 2963us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:26.679304 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:26.679304 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:26.679313 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:26.681336 (+ 2023us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.681754 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.681756 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.681756 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.681786 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:26.682021 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:26.682022 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:26.682022 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:26.682158 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:26.682256 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:26.682677 (+ 421us) server_negotiation.cc:300] Negotiation successful
1023 20:15:26.682772 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:15:26.682850 24025 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:26.683588 24025 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:26.685187 23865 raft_consensus.cc:1275] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [term 1 FOLLOWER]: Refusing update from remote peer 743e2243be854fb391ab804650234e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:26.685182 23788 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [term 1 FOLLOWER]: Refusing update from remote peer 743e2243be854fb391ab804650234e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251023 20:15:26.685448 24022 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:15:26.685587 24022 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:15:26.685568 23974 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.685734 24023 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.687028 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743e2243be854fb391ab804650234e3f. Latest consensus state: current_term: 1 leader_uuid: "743e2243be854fb391ab804650234e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.687072 23998 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743e2243be854fb391ab804650234e3f. Latest consensus state: current_term: 1 leader_uuid: "743e2243be854fb391ab804650234e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.687142 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:26.687150 23998 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:26.688244 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "743e2243be854fb391ab804650234e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.688283 23998 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "743e2243be854fb391ab804650234e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.688351 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:26.688354 23998 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:26.688982 24026 mvcc.cc:204] Tried to move back new op lower bound from 7214082157299814400 to 7214082157275750400. Current Snapshot: MvccSnapshot[applied={T|T < 7214082157299814400}]
I20251023 20:15:26.689364 24029 mvcc.cc:204] Tried to move back new op lower bound from 7214082157299814400 to 7214082157275750400. Current Snapshot: MvccSnapshot[applied={T|T < 7214082157299814400}]
I20251023 20:15:26.690546 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "743e2243be854fb391ab804650234e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.690654 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:26.691004 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "743e2243be854fb391ab804650234e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9544b1f0e6444a98b98ada8dd3761e1c" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33327 } } peers { permanent_uuid: "cae26edcaf494f5c9570b7de5aac2021" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39265 } } peers { permanent_uuid: "743e2243be854fb391ab804650234e3f" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 42197 } } }
I20251023 20:15:26.691093 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:26.692003 24028 mvcc.cc:204] Tried to move back new op lower bound from 7214082157299814400 to 7214082157275750400. Current Snapshot: MvccSnapshot[applied={T|T < 7214082157299814400 or (T in {7214082157299814400})}]
I20251023 20:15:26.693249 24025 catalog_manager.cc:1357] Generated new cluster ID: 1b1da79eff0840dba3a16ae2fa2945b0
I20251023 20:15:26.693313 24025 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:15:26.713994 24025 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:15:26.715178 24025 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:15:26.726142 24025 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 743e2243be854fb391ab804650234e3f: Generated new TSK 0
I20251023 20:15:26.726562 24025 catalog_manager.cc:1524] Initializing in-progress tserver states...
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.9.117.252@KRBTEST.COM
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.9.117.254@KRBTEST.COM
Oct 23 20:15:26 dist-test-slave-x62p krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.9.117.253@KRBTEST.COM
I20251023 20:15:27.670801 24009 catalog_manager.cc:1269] Loaded cluster ID: 1b1da79eff0840dba3a16ae2fa2945b0
I20251023 20:15:27.670917 24009 catalog_manager.cc:1562] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021: loading cluster ID for follower catalog manager: success
I20251023 20:15:27.671891 24009 catalog_manager.cc:1584] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021: acquiring CA information for follower catalog manager: success
I20251023 20:15:27.672243 24009 catalog_manager.cc:1612] T 00000000000000000000000000000000 P cae26edcaf494f5c9570b7de5aac2021: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:15:27.686233 24022 catalog_manager.cc:1269] Loaded cluster ID: 1b1da79eff0840dba3a16ae2fa2945b0
I20251023 20:15:27.686338 24022 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c: loading cluster ID for follower catalog manager: success
I20251023 20:15:27.687275 24022 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c: acquiring CA information for follower catalog manager: success
I20251023 20:15:27.687602 24022 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 9544b1f0e6444a98b98ada8dd3761e1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:15:27.871531 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 23746
I20251023 20:15:27.877728 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 23823
I20251023 20:15:27.883487 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 23900
2025-10-23T20:15:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3990 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:15:27 dist-test-slave-x62p krb5kdc[24042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:15:27 dist-test-slave-x62p krb5kdc[24042](info): set up 2 sockets
Oct 23 20:15:27 dist-test-slave-x62p krb5kdc[24042](info): commencing operation
krb5kdc: starting...
W20251023 20:15:29.901682 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.996s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:15:29 dist-test-slave-x62p krb5kdc[24042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:15:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:15:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:30.047797 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:34751
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:34751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:30.132807 24058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:30.133025 24058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:30.133051 24058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:30.134680 24058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:30.134735 24058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:30.134750 24058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:30.134764 24058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:30.134778 24058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:30.136564 24058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:34751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:34751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:30.136863 24058 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:30.137058 24058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:30.140091 24064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:30.140239 24058 server_base.cc:1047] running on GCE node
W20251023 20:15:30.140077 24063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:30.140077 24066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:30.140604 24058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:30.141007 24058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:30.142163 24058 hybrid_clock.cc:648] HybridClock initialized: now 1761250530142146 us; error 40 us; skew 500 ppm
Oct 23 20:15:30 dist-test-slave-x62p krb5kdc[24042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250530, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:30.145310 24058 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:15:30.146250 24058 webserver.cc:492] Webserver started at http://127.9.117.254:35017/ using document root <none> and password file <none>
I20251023 20:15:30.146530 24058 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:30.146598 24058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:30.146763 24058 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:30.148391 24058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "5cf4496fafe54de885f5588850581878"
format_stamp: "Formatted at 2025-10-23 20:15:30 on dist-test-slave-x62p"
server_key: "c4aebb3005c705c914da2e4cf12dbd1a"
server_key_iv: "a92e656a6ac3549cfa42ca8d760cbdf6"
server_key_version: "encryptionkey@0"
I20251023 20:15:30.148916 24058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "5cf4496fafe54de885f5588850581878"
format_stamp: "Formatted at 2025-10-23 20:15:30 on dist-test-slave-x62p"
server_key: "c4aebb3005c705c914da2e4cf12dbd1a"
server_key_iv: "a92e656a6ac3549cfa42ca8d760cbdf6"
server_key_version: "encryptionkey@0"
I20251023 20:15:30.150782 24058 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:30.152189 24073 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:30.152426 24058 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:30.152521 24058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "5cf4496fafe54de885f5588850581878"
format_stamp: "Formatted at 2025-10-23 20:15:30 on dist-test-slave-x62p"
server_key: "c4aebb3005c705c914da2e4cf12dbd1a"
server_key_iv: "a92e656a6ac3549cfa42ca8d760cbdf6"
server_key_version: "encryptionkey@0"
I20251023 20:15:30.152614 24058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:30.165638 24058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:30.173432 24058 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:30.174038 24058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:30.178870 24058 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:34751
I20251023 20:15:30.178920 24125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:34751 every 8 connection(s)
I20251023 20:15:30.179271 24058 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:30.179994 24126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.183055 24126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878: Bootstrap starting.
I20251023 20:15:30.183084 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 24058
I20251023 20:15:30.183198 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:15:30.183383 9687 external_mini_cluster.cc:1468] Setting key ee84911a2fed2fe33ef00466db079730
I20251023 20:15:30.183934 24126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.184267 24126 log.cc:826] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:30.185318 24126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878: No bootstrap required, opened a new log
I20251023 20:15:30.186751 24126 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf4496fafe54de885f5588850581878" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34751 } }
I20251023 20:15:30.186908 24126 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.186931 24126 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf4496fafe54de885f5588850581878, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.187043 24126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf4496fafe54de885f5588850581878" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34751 } }
I20251023 20:15:30.187119 24126 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:15:30.187148 24126 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:15:30.187196 24126 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.188100 24126 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf4496fafe54de885f5588850581878" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34751 } }
I20251023 20:15:30.188235 24126 leader_election.cc:304] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf4496fafe54de885f5588850581878; no voters:
I20251023 20:15:30.188431 24126 leader_election.cc:290] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:15:30.188494 24131 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.188663 24131 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [term 1 LEADER]: Becoming Leader. State: Replica: 5cf4496fafe54de885f5588850581878, State: Running, Role: LEADER
I20251023 20:15:30.188759 24126 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:30.188828 24131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf4496fafe54de885f5588850581878" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34751 } }
I20251023 20:15:30.189196 24131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf4496fafe54de885f5588850581878. Latest consensus state: current_term: 1 leader_uuid: "5cf4496fafe54de885f5588850581878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf4496fafe54de885f5588850581878" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34751 } } }
I20251023 20:15:30.189301 24133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf4496fafe54de885f5588850581878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf4496fafe54de885f5588850581878" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 34751 } } }
I20251023 20:15:30.189378 24133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:30.189312 24131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:30.189656 24137 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
Oct 23 20:15:30 dist-test-slave-x62p krb5kdc[24042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:30.190382 24137 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:30.192209 24137 catalog_manager.cc:1357] Generated new cluster ID: 2f760d331cde4b52a3caf3d6f7f6ff1c
I20251023 20:15:30.192269 24137 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:15:30.194945 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:30.183987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.9.117.254:34751)
1023 20:15:30.184171 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:30.184182 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:30.184206 (+ 24us) server_negotiation.cc:408] Connection header received
1023 20:15:30.184713 (+ 507us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:30.184751 (+ 38us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:30.184829 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:30.185000 (+ 171us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:30.186090 (+ 1090us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:30.186871 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:30.187753 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:30.187984 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:30.190945 (+ 2961us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:30.190954 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:30.190954 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:30.190985 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:30.193203 (+ 2218us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:30.193773 (+ 570us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:30.193776 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:30.193776 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:30.193852 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:30.194035 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:30.194035 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:30.194036 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:30.194290 (+ 254us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:30.194436 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:30.194628 (+ 192us) server_negotiation.cc:300] Negotiation successful
1023 20:15:30.194757 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20251023 20:15:30.196089 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:15:30.207314 24137 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:15:30.208175 24137 catalog_manager.cc:1514] Loading token signing keys...
Oct 23 20:15:30 dist-test-slave-x62p krb5kdc[24042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20251023 20:15:30.223069 24137 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5cf4496fafe54de885f5588850581878: Generated new TSK 0
I20251023 20:15:30.223294 24137 catalog_manager.cc:1524] Initializing in-progress tserver states...
Oct 23 20:15:30 dist-test-slave-x62p krb5kdc[24042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.9.117.254@KRBTEST.COM
I20251023 20:15:30.283162 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 24058
2025-10-23T20:15:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2397 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:15:30 dist-test-slave-x62p krb5kdc[24155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:15:30 dist-test-slave-x62p krb5kdc[24155](info): set up 2 sockets
Oct 23 20:15:30 dist-test-slave-x62p krb5kdc[24155](info): commencing operation
krb5kdc: starting...
W20251023 20:15:32.321350 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:15:32 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:15:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:32.469053 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43961
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:32.558856 24171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:32.559078 24171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:32.559116 24171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:32.560943 24171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:32.561030 24171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:32.561064 24171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:32.561085 24171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:32.561103 24171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:32.563311 24171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:43961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:32.563668 24171 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:32.563898 24171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:32.566924 24177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:32.566854 24176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:32.566859 24179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:32.567114 24171 server_base.cc:1047] running on GCE node
I20251023 20:15:32.567414 24171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:32.567714 24171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:32.568864 24171 hybrid_clock.cc:648] HybridClock initialized: now 1761250532568842 us; error 37 us; skew 500 ppm
Oct 23 20:15:32 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250532, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:32.571703 24171 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:15:32.572368 24171 webserver.cc:492] Webserver started at http://127.9.117.254:36875/ using document root <none> and password file <none>
I20251023 20:15:32.572583 24171 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:32.572626 24171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:32.572765 24171 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:32.573931 24171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "bd0af698e2ba42349d4a183da8ef3125"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "4a77ecdb4e9236604d46ecc03871d2db"
server_key_iv: "1498cc9261714ab3baa7617944d948a7"
server_key_version: "encryptionkey@0"
I20251023 20:15:32.574307 24171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "bd0af698e2ba42349d4a183da8ef3125"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "4a77ecdb4e9236604d46ecc03871d2db"
server_key_iv: "1498cc9261714ab3baa7617944d948a7"
server_key_version: "encryptionkey@0"
I20251023 20:15:32.575820 24171 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:15:32.576870 24186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:32.577075 24171 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:32.577168 24171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "bd0af698e2ba42349d4a183da8ef3125"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "4a77ecdb4e9236604d46ecc03871d2db"
server_key_iv: "1498cc9261714ab3baa7617944d948a7"
server_key_version: "encryptionkey@0"
I20251023 20:15:32.577250 24171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:32.593006 24171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:32.595352 24171 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:32.595526 24171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:32.599921 24171 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:43961
I20251023 20:15:32.599963 24238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:43961 every 8 connection(s)
I20251023 20:15:32.600346 24171 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:32.601080 24239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.603950 24239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Bootstrap starting.
I20251023 20:15:32.604631 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 24171
I20251023 20:15:32.604668 24239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.604794 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:15:32.605026 9687 external_mini_cluster.cc:1468] Setting key 605dc6f164b81c4a676cc6ea125bf8f1
I20251023 20:15:32.605064 24239 log.cc:826] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:32.605952 24239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: No bootstrap required, opened a new log
I20251023 20:15:32.608234 24239 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:32.608387 24239 raft_consensus.cc:385] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.608409 24239 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0af698e2ba42349d4a183da8ef3125, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.608508 24239 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:32.608642 24239 raft_consensus.cc:399] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:15:32.608671 24239 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:15:32.608712 24239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.609688 24239 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:32.609849 24239 leader_election.cc:304] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd0af698e2ba42349d4a183da8ef3125; no voters:
I20251023 20:15:32.610064 24239 leader_election.cc:290] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:15:32.610399 24239 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:32.611037 24244 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.611269 24244 raft_consensus.cc:697] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 1 LEADER]: Becoming Leader. State: Replica: bd0af698e2ba42349d4a183da8ef3125, State: Running, Role: LEADER
I20251023 20:15:32.611449 24244 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:32.611897 24248 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0af698e2ba42349d4a183da8ef3125. Latest consensus state: current_term: 1 leader_uuid: "bd0af698e2ba42349d4a183da8ef3125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } } }
I20251023 20:15:32.611989 24248 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:32.612553 24245 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd0af698e2ba42349d4a183da8ef3125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } } }
I20251023 20:15:32.612632 24245 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:32.612789 24253 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
Oct 23 20:15:32 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:32.613701 24253 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:32.615444 24253 catalog_manager.cc:1357] Generated new cluster ID: 0fcf776b6f584b83b1ab2c0eff389219
I20251023 20:15:32.615501 24253 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:15:32.618578 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:32.605834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.9.117.254:43961)
1023 20:15:32.606019 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:32.606033 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:32.606060 (+ 27us) server_negotiation.cc:408] Connection header received
1023 20:15:32.606463 (+ 403us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:32.606469 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:32.606524 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:32.606647 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:32.607845 (+ 1198us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:32.609149 (+ 1304us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:32.610151 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:32.610655 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:32.614024 (+ 3369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:32.614033 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:32.614033 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:32.614060 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:32.616799 (+ 2739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:32.617279 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:32.617283 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:32.617283 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:32.617358 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:32.617672 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:32.617673 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:32.617673 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:32.617932 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:32.618062 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:32.618232 (+ 170us) server_negotiation.cc:300] Negotiation successful
1023 20:15:32.618363 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20251023 20:15:32.621680 24253 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:15:32.622325 24253 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:15:32.628130 24253 catalog_manager.cc:6022] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Generated new TSK 0
I20251023 20:15:32.628367 24253 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:32.692443 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43961
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:32.788441 24267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:32.788663 24267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:32.788692 24267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:32.790377 24267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:32.790437 24267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:32.790470 24267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:15:32.792282 24267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:32.792551 24267 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:32.792820 24267 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:32.795517 24272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:32.795536 24273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:32.795501 24275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:32.795658 24267 server_base.cc:1047] running on GCE node
I20251023 20:15:32.796039 24267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:32.796396 24267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:32.797564 24267 hybrid_clock.cc:648] HybridClock initialized: now 1761250532797537 us; error 54 us; skew 500 ppm
Oct 23 20:15:32 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250532, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:32.801980 24267 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:15:32.803038 24267 webserver.cc:492] Webserver started at http://127.9.117.193:37277/ using document root <none> and password file <none>
I20251023 20:15:32.803393 24267 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:32.803483 24267 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:32.803695 24267 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:32.805384 24267 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "4315e7383e354eb09243f189ed5f606e"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "c9f9a226b6e9cca12ee161e8ba6daf08"
server_key_iv: "b8b41351d0884f47ce6ab5143a6d63cf"
server_key_version: "encryptionkey@0"
I20251023 20:15:32.805892 24267 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "4315e7383e354eb09243f189ed5f606e"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "c9f9a226b6e9cca12ee161e8ba6daf08"
server_key_iv: "b8b41351d0884f47ce6ab5143a6d63cf"
server_key_version: "encryptionkey@0"
I20251023 20:15:32.808100 24267 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251023 20:15:32.809450 24282 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:32.809805 24267 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:32.809962 24267 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "4315e7383e354eb09243f189ed5f606e"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "c9f9a226b6e9cca12ee161e8ba6daf08"
server_key_iv: "b8b41351d0884f47ce6ab5143a6d63cf"
server_key_version: "encryptionkey@0"
I20251023 20:15:32.810083 24267 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:32.830322 24267 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:32.833045 24267 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:32.833185 24267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:32.833400 24267 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:32.833920 24267 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:32.833973 24267 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:32.833998 24267 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:32.834012 24267 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:32.841328 24267 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:37585
I20251023 20:15:32.841452 24395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:37585 every 8 connection(s)
I20251023 20:15:32.841755 24267 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:15:32.848475 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 24267
Oct 23 20:15:32 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250532, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:32.848618 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:15:32.848868 9687 external_mini_cluster.cc:1468] Setting key e3d3880c9cc3e68b04cb4bc290478522
I20251023 20:15:32.853106 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:32.842756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:59331 (local address 127.9.117.254:43961)
1023 20:15:32.842820 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:32.842822 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:32.843008 (+ 186us) server_negotiation.cc:408] Connection header received
1023 20:15:32.843600 (+ 592us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:32.843602 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:32.843614 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:32.843692 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:32.844669 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:32.845253 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:32.845941 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:32.846085 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:32.849385 (+ 3300us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:32.849391 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:32.849392 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:32.849400 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:32.851100 (+ 1700us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:32.851714 (+ 614us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:32.851715 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:32.851716 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:32.851756 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:32.852219 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:32.852220 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:32.852221 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:32.852348 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:32.852422 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:32.852914 (+ 492us) server_negotiation.cc:300] Negotiation successful
1023 20:15:32.853022 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:15:32.853395 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:32.842682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:59331)
1023 20:15:32.842853 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:32.842890 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:32.843452 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:32.843806 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:32.843811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:32.844183 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:32.844508 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:32.844512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:32.845363 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:32.845364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:32.845800 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:32.845802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:32.846067 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:32.847243 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:32.847267 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:32.849279 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:32.851210 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:32.851223 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:32.851227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:32.851608 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:32.851867 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:32.851868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:32.851869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:32.851992 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:32.852431 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:32.852440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:32.852677 (+ 237us) client_negotiation.cc:769] Sending connection context
1023 20:15:32.852860 (+ 183us) client_negotiation.cc:240] Negotiation successful
1023 20:15:32.853090 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20251023 20:15:32.854146 24396 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43961
I20251023 20:15:32.854261 24396 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:32.854503 24396 heartbeater.cc:507] Master 127.9.117.254:43961 requested a full tablet report, sending...
I20251023 20:15:32.855095 24203 ts_manager.cc:194] Registered new tserver with Master: 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585)
I20251023 20:15:32.855957 24203 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:59331
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:32.913637 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43961
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:33.020259 24403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:33.020493 24403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:33.020547 24403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:33.023008 24403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:33.023106 24403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:33.023173 24403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:15:33.025328 24403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:33.025669 24403 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:33.025895 24403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:33.029116 24408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:33.029170 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:33.029218 24409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:33.029294 24403 server_base.cc:1047] running on GCE node
I20251023 20:15:33.029567 24403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:33.029845 24403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:33.030999 24403 hybrid_clock.cc:648] HybridClock initialized: now 1761250533030979 us; error 38 us; skew 500 ppm
Oct 23 20:15:33 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:33.033910 24403 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:15:33.034559 24403 webserver.cc:492] Webserver started at http://127.9.117.194:41595/ using document root <none> and password file <none>
I20251023 20:15:33.034775 24403 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:33.034813 24403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:33.034927 24403 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:33.035934 24403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "9348e9ed0255496dbd6e8791382b7a07"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "82aa03ac8b230460ab61322eddfb61a5"
server_key_iv: "635c86c987d2dbdc83efc60f16375257"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.036264 24403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "9348e9ed0255496dbd6e8791382b7a07"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "82aa03ac8b230460ab61322eddfb61a5"
server_key_iv: "635c86c987d2dbdc83efc60f16375257"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.037662 24403 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:15:33.038467 24418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.038713 24403 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:33.038812 24403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "9348e9ed0255496dbd6e8791382b7a07"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "82aa03ac8b230460ab61322eddfb61a5"
server_key_iv: "635c86c987d2dbdc83efc60f16375257"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.038887 24403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:33.076377 24403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:33.079000 24403 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:33.079169 24403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:33.079421 24403 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:33.079756 24403 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:33.079792 24403 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.079829 24403 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:33.079857 24403 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.086341 24403 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:42639
I20251023 20:15:33.086434 24531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:42639 every 8 connection(s)
I20251023 20:15:33.086777 24403 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:15:33.090801 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 24403
I20251023 20:15:33.090894 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:15:33.091073 9687 external_mini_cluster.cc:1468] Setting key a8802986a1092e4a814b1804f7d14b8f
Oct 23 20:15:33 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:33.098326 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.088063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:49125 (local address 127.9.117.254:43961)
1023 20:15:33.088161 (+ 98us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.088164 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.088316 (+ 152us) server_negotiation.cc:408] Connection header received
1023 20:15:33.088940 (+ 624us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.088942 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.088955 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.089017 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:33.089882 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.090404 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.091072 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.091251 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.094433 (+ 3182us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:33.094438 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:33.094439 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:33.094449 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:33.096463 (+ 2014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:33.097190 (+ 727us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:33.097192 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:33.097193 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:33.097228 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:33.097483 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:33.097485 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:33.097485 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:33.097661 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:33.097750 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.098090 (+ 340us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.098200 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251023 20:15:33.098343 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.087968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:49125)
1023 20:15:33.088164 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.088198 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.088812 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.089084 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.089089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.089457 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:33.089773 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.089777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.090498 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.090499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.090988 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.090991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.091146 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.092270 (+ 1124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:33.092289 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:33.094307 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:33.096624 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:33.096636 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:33.096640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:33.097103 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:33.097303 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:33.097304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:33.097305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:33.097364 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:33.097761 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:33.097767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:33.097921 (+ 154us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.098029 (+ 108us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.098163 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20251023 20:15:33.099182 24532 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43961
I20251023 20:15:33.099306 24532 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:33.099612 24532 heartbeater.cc:507] Master 127.9.117.254:43961 requested a full tablet report, sending...
I20251023 20:15:33.100142 24203 ts_manager.cc:194] Registered new tserver with Master: 9348e9ed0255496dbd6e8791382b7a07 (127.9.117.194:42639)
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
I20251023 20:15:33.100597 24203 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:49125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:33.148628 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:43961
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:33.257076 24539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:33.257452 24539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:33.257500 24539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:33.259917 24539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:33.259996 24539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:33.260033 24539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:15:33.262096 24539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:33.262385 24539 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:33.262598 24539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:33.265486 24545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:33.265491 24544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:33.265486 24547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:33.265815 24539 server_base.cc:1047] running on GCE node
I20251023 20:15:33.266119 24539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:33.266397 24539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:33.267565 24539 hybrid_clock.cc:648] HybridClock initialized: now 1761250533267546 us; error 43 us; skew 500 ppm
Oct 23 20:15:33 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:33.270325 24539 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:15:33.270887 24539 webserver.cc:492] Webserver started at http://127.9.117.195:40319/ using document root <none> and password file <none>
I20251023 20:15:33.271075 24539 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:33.271111 24539 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:33.271200 24539 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:33.272207 24539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "f73de5713f8f45968cc13181138452b0"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "17df9c66e5711382e062c974866fbdc2"
server_key_iv: "db2e7a90e1e22d9d5dacfa1e992daed8"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.272554 24539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "f73de5713f8f45968cc13181138452b0"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "17df9c66e5711382e062c974866fbdc2"
server_key_iv: "db2e7a90e1e22d9d5dacfa1e992daed8"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.273970 24539 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:15:33.274755 24554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.274942 24539 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:33.275022 24539 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "f73de5713f8f45968cc13181138452b0"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "17df9c66e5711382e062c974866fbdc2"
server_key_iv: "db2e7a90e1e22d9d5dacfa1e992daed8"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.275096 24539 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:33.290831 24539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:33.293916 24539 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:33.294112 24539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:33.294385 24539 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:33.294778 24539 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:33.294816 24539 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.294854 24539 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:33.294888 24539 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.301574 24539 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:44921
I20251023 20:15:33.301645 24667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:44921 every 8 connection(s)
I20251023 20:15:33.301976 24539 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:15:33.305692 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 24539
I20251023 20:15:33.305788 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:15:33.305979 9687 external_mini_cluster.cc:1468] Setting key 3df5b64ccf5b39a8ca48e35eac4597e8
Oct 23 20:15:33 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:33.312326 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.302774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:47811 (local address 127.9.117.254:43961)
1023 20:15:33.302890 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.302893 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.303059 (+ 166us) server_negotiation.cc:408] Connection header received
1023 20:15:33.303695 (+ 636us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.303697 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.303712 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.303779 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:33.304678 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.305311 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.305970 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.306150 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.309092 (+ 2942us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:33.309103 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:33.309103 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:33.309112 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:33.310661 (+ 1549us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:33.311215 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:33.311216 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:33.311216 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:33.311246 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:33.311513 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:33.311514 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:33.311515 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:33.311677 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:33.311778 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.312159 (+ 381us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.312252 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251023 20:15:33.312477 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.302718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:47811)
1023 20:15:33.302941 (+ 223us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.302973 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.303586 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.303864 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.303869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.304237 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:33.304567 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.304571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.305410 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.305411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.305883 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.305885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.306057 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.307306 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:33.307329 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:33.308951 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:33.310779 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:33.310845 (+ 66us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:33.310850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:33.311112 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:33.311316 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:33.311317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:33.311317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:33.311422 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:33.311812 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:33.311822 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:33.311995 (+ 173us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.312109 (+ 114us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.312253 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20251023 20:15:33.313252 24668 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43961
I20251023 20:15:33.313372 24668 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:33.313616 24668 heartbeater.cc:507] Master 127.9.117.254:43961 requested a full tablet report, sending...
I20251023 20:15:33.314211 24203 ts_manager.cc:194] Registered new tserver with Master: f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921)
I20251023 20:15:33.314666 24203 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:47811
I20251023 20:15:33.318300 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:15:33.326241 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.319852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.9.117.254:43961)
1023 20:15:33.319960 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.319963 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.320044 (+ 81us) server_negotiation.cc:408] Connection header received
1023 20:15:33.320126 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.320127 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.320139 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.320194 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:33.321073 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.321579 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.322165 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.322365 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.323499 (+ 1134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:33.323503 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:33.323504 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:33.323512 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:33.325132 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:33.325465 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:33.325466 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:33.325466 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:33.325504 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:33.325650 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:33.325651 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:33.325651 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:33.325798 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:33.325928 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.326036 (+ 108us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.326132 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251023 20:15:33.327970 24203 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:33.329398 24203 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:33.335752 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.332070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:44921 (local address 127.0.0.1:44712)
1023 20:15:33.332219 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.332229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.332295 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.332983 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.332986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.332989 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:33.333295 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.333298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.334729 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.334730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.335519 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.335521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.335582 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.335584 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.335614 (+ 30us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.335645 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":108,"threads_started":1}
I20251023 20:15:33.336185 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.332296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:37585 (local address 127.0.0.1:35230)
1023 20:15:33.332431 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.332441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.332490 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.332992 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.332993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.332997 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:33.333323 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.333325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.334466 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.334468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.335362 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.335365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.335445 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.335499 (+ 54us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.335553 (+ 54us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.335639 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20251023 20:15:33.336329 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.332533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.9.117.193:37585)
1023 20:15:33.332715 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.332755 (+ 40us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.332775 (+ 20us) server_negotiation.cc:408] Connection header received
1023 20:15:33.332833 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.332835 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.332858 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.333418 (+ 560us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:33.333522 (+ 104us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.334359 (+ 837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.335476 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.336093 (+ 617us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.336161 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.336197 (+ 36us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.336249 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20251023 20:15:33.336620 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.332316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.9.117.194:42639)
1023 20:15:33.332893 (+ 577us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.332901 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.332912 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:15:33.332958 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.332961 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.332979 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.333053 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:33.333496 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.334654 (+ 1158us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.335819 (+ 1165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.336379 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.336435 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.336471 (+ 36us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.336530 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":127,"threads_started":1}
I20251023 20:15:33.335752 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.332047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42639 (local address 127.0.0.1:58398)
1023 20:15:33.332176 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.332209 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.332295 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.333201 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.333202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.333205 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:33.333390 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.333391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.334858 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.334860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.335488 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.335491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.335568 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.335573 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.335618 (+ 45us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.335662 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20251023 20:15:33.337160 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.332564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.9.117.195:44921)
1023 20:15:33.332763 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.332772 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.332783 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:15:33.332833 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.332836 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.332857 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.333355 (+ 498us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:33.333444 (+ 89us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.334635 (+ 1191us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.336176 (+ 1541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.336927 (+ 751us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.336980 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.337015 (+ 35us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.337073 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20251023 20:15:33.337116 24330 tablet_service.cc:1505] Processing CreateTablet for tablet cbf88a308d6b41fdb6cc2397c95b7612 (DEFAULT_TABLE table=test-table [id=d8a6ff01bf494d25b14e4283a8a67381]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:33.337464 24330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf88a308d6b41fdb6cc2397c95b7612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:33.337455 24466 tablet_service.cc:1505] Processing CreateTablet for tablet cbf88a308d6b41fdb6cc2397c95b7612 (DEFAULT_TABLE table=test-table [id=d8a6ff01bf494d25b14e4283a8a67381]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:33.337746 24466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf88a308d6b41fdb6cc2397c95b7612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:33.337777 24602 tablet_service.cc:1505] Processing CreateTablet for tablet cbf88a308d6b41fdb6cc2397c95b7612 (DEFAULT_TABLE table=test-table [id=d8a6ff01bf494d25b14e4283a8a67381]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:33.338032 24602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf88a308d6b41fdb6cc2397c95b7612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:33.340163 24685 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Bootstrap starting.
I20251023 20:15:33.340659 24686 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: Bootstrap starting.
I20251023 20:15:33.340792 24685 tablet_bootstrap.cc:654] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.341147 24685 log.cc:826] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:33.341157 24687 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: Bootstrap starting.
I20251023 20:15:33.341336 24686 tablet_bootstrap.cc:654] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.341707 24687 tablet_bootstrap.cc:654] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.341710 24686 log.cc:826] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:33.342058 24685 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: No bootstrap required, opened a new log
I20251023 20:15:33.342046 24687 log.cc:826] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:33.342123 24685 ts_tablet_manager.cc:1403] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:33.342381 24686 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: No bootstrap required, opened a new log
I20251023 20:15:33.342443 24686 ts_tablet_manager.cc:1403] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:33.342834 24687 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: No bootstrap required, opened a new log
I20251023 20:15:33.342892 24687 ts_tablet_manager.cc:1403] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:33.343823 24685 raft_consensus.cc:359] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.343982 24685 raft_consensus.cc:385] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.344012 24685 raft_consensus.cc:740] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4315e7383e354eb09243f189ed5f606e, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.344061 24686 raft_consensus.cc:359] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.344162 24686 raft_consensus.cc:385] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.344112 24685 consensus_queue.cc:260] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.344180 24686 raft_consensus.cc:740] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9348e9ed0255496dbd6e8791382b7a07, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.344255 24686 consensus_queue.cc:260] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.344386 24685 ts_tablet_manager.cc:1434] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:33.344504 24532 heartbeater.cc:499] Master 127.9.117.254:43961 was elected leader, sending a full tablet report...
I20251023 20:15:33.344596 24396 heartbeater.cc:499] Master 127.9.117.254:43961 was elected leader, sending a full tablet report...
I20251023 20:15:33.344581 24687 raft_consensus.cc:359] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.344712 24687 raft_consensus.cc:385] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.344770 24687 raft_consensus.cc:740] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73de5713f8f45968cc13181138452b0, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.344867 24687 consensus_queue.cc:260] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.345194 24686 ts_tablet_manager.cc:1434] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:33.345402 24687 ts_tablet_manager.cc:1434] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251023 20:15:33.345419 24668 heartbeater.cc:499] Master 127.9.117.254:43961 was elected leader, sending a full tablet report...
I20251023 20:15:33.453820 24692 raft_consensus.cc:493] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.453991 24692 raft_consensus.cc:515] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.454420 24692 leader_election.cc:290] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921), 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585)
I20251023 20:15:33.456955 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.454583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:44921 (local address 127.9.117.194:39703)
1023 20:15:33.454671 (+ 88us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.454686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.454745 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.454966 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.454967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.454970 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:33.455244 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.455246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.456067 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.456068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.456692 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.456694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.456811 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.456816 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.456842 (+ 26us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.456880 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20251023 20:15:33.457065 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.454682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:37585 (local address 127.9.117.194:36041)
1023 20:15:33.454893 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.454903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.454974 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.455173 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.455174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.455177 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:33.455468 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.455470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.456161 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.456162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.456817 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.456819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.456904 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.456907 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.456940 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.456977 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":127,"threads_started":1}
I20251023 20:15:33.457484 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.454745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:39703 (local address 127.9.117.195:44921)
1023 20:15:33.454807 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.454810 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.454819 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:15:33.454856 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.454858 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.454871 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.454956 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:33.455358 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.455981 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.456825 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.457355 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.457365 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.457388 (+ 23us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.457415 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251023 20:15:33.457537 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.454778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:36041 (local address 127.9.117.193:37585)
1023 20:15:33.454829 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.454831 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.454980 (+ 149us) server_negotiation.cc:408] Connection header received
1023 20:15:33.455042 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.455043 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.455062 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.455132 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:33.455559 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.456060 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.456919 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.457407 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.457434 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.457457 (+ 23us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.457485 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:15:33.457855 24622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbf88a308d6b41fdb6cc2397c95b7612" candidate_uuid: "9348e9ed0255496dbd6e8791382b7a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f73de5713f8f45968cc13181138452b0" is_pre_election: true
I20251023 20:15:33.457870 24350 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbf88a308d6b41fdb6cc2397c95b7612" candidate_uuid: "9348e9ed0255496dbd6e8791382b7a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4315e7383e354eb09243f189ed5f606e" is_pre_election: true
I20251023 20:15:33.458006 24350 raft_consensus.cc:2468] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9348e9ed0255496dbd6e8791382b7a07 in term 0.
I20251023 20:15:33.458015 24622 raft_consensus.cc:2468] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9348e9ed0255496dbd6e8791382b7a07 in term 0.
I20251023 20:15:33.458259 24420 leader_election.cc:304] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4315e7383e354eb09243f189ed5f606e, 9348e9ed0255496dbd6e8791382b7a07; no voters:
I20251023 20:15:33.458392 24692 raft_consensus.cc:2804] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.458459 24692 raft_consensus.cc:493] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.458488 24692 raft_consensus.cc:3060] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.459189 24692 raft_consensus.cc:515] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.459319 24692 leader_election.cc:290] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 1 election: Requested vote from peers f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921), 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585)
I20251023 20:15:33.459542 24622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbf88a308d6b41fdb6cc2397c95b7612" candidate_uuid: "9348e9ed0255496dbd6e8791382b7a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f73de5713f8f45968cc13181138452b0"
I20251023 20:15:33.459561 24350 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbf88a308d6b41fdb6cc2397c95b7612" candidate_uuid: "9348e9ed0255496dbd6e8791382b7a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4315e7383e354eb09243f189ed5f606e"
I20251023 20:15:33.459640 24350 raft_consensus.cc:3060] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.459640 24622 raft_consensus.cc:3060] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.460412 24350 raft_consensus.cc:2468] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9348e9ed0255496dbd6e8791382b7a07 in term 1.
I20251023 20:15:33.460457 24622 raft_consensus.cc:2468] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9348e9ed0255496dbd6e8791382b7a07 in term 1.
I20251023 20:15:33.460646 24420 leader_election.cc:304] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9348e9ed0255496dbd6e8791382b7a07, f73de5713f8f45968cc13181138452b0; no voters:
I20251023 20:15:33.460757 24692 raft_consensus.cc:2804] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.460924 24692 raft_consensus.cc:697] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 LEADER]: Becoming Leader. State: Replica: 9348e9ed0255496dbd6e8791382b7a07, State: Running, Role: LEADER
I20251023 20:15:33.461045 24692 consensus_queue.cc:237] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.461967 24201 catalog_manager.cc:5649] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9348e9ed0255496dbd6e8791382b7a07 (127.9.117.194). New cstate: current_term: 1 leader_uuid: "9348e9ed0255496dbd6e8791382b7a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:33.488798 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.486362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.9.117.194:42639)
1023 20:15:33.486446 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.486448 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.486458 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:15:33.486508 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.486509 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.486528 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.486592 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:15:33.486862 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.487390 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.488093 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.488263 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.488313 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:15:33.488436 (+ 123us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:15:33.488512 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.488607 (+ 95us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.488642 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251023 20:15:33.490810 24350 raft_consensus.cc:1275] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Refusing update from remote peer 9348e9ed0255496dbd6e8791382b7a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:33.490811 24622 raft_consensus.cc:1275] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 1 FOLLOWER]: Refusing update from remote peer 9348e9ed0255496dbd6e8791382b7a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:33.491209 24695 consensus_queue.cc:1048] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.491345 24692 consensus_queue.cc:1048] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.493909 24698 mvcc.cc:204] Tried to move back new op lower bound from 7214082185175293952 to 7214082185057095680. Current Snapshot: MvccSnapshot[applied={T|T < 7214082185175293952}]
I20251023 20:15:33.494089 24699 mvcc.cc:204] Tried to move back new op lower bound from 7214082185175293952 to 7214082185057095680. Current Snapshot: MvccSnapshot[applied={T|T < 7214082185175293952}]
I20251023 20:15:33.494314 24700 mvcc.cc:204] Tried to move back new op lower bound from 7214082185175293952 to 7214082185057095680. Current Snapshot: MvccSnapshot[applied={T|T < 7214082185175293952}]
I20251023 20:15:33.497094 24201 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51590:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:15:33.497198 24201 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51590:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:15:33.498026 24201 catalog_manager.cc:5936] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Sending DeleteTablet for 3 replicas of tablet cbf88a308d6b41fdb6cc2397c95b7612
I20251023 20:15:33.498415 24466 tablet_service.cc:1552] Processing DeleteTablet for tablet cbf88a308d6b41fdb6cc2397c95b7612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:33 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:58398
I20251023 20:15:33.498415 24330 tablet_service.cc:1552] Processing DeleteTablet for tablet cbf88a308d6b41fdb6cc2397c95b7612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:33 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:35230
I20251023 20:15:33.498689 24708 tablet_replica.cc:333] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: stopping tablet replica
I20251023 20:15:33.498698 24602 tablet_service.cc:1552] Processing DeleteTablet for tablet cbf88a308d6b41fdb6cc2397c95b7612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:33 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:44712
I20251023 20:15:33.498790 24708 raft_consensus.cc:2243] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:33.498909 24708 raft_consensus.cc:2272] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.498955 24710 tablet_replica.cc:333] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: stopping tablet replica
I20251023 20:15:33.499009 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 24267
I20251023 20:15:33.499070 24709 tablet_replica.cc:333] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: stopping tablet replica
I20251023 20:15:33.499147 24709 raft_consensus.cc:2243] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:33.499269 24709 raft_consensus.cc:2272] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.499665 24710 raft_consensus.cc:2243] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:33.499786 24710 raft_consensus.cc:2272] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.499882 24709 ts_tablet_manager.cc:1916] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.500240 24710 ts_tablet_manager.cc:1916] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.502120 24709 ts_tablet_manager.cc:1929] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:33.502197 24709 log.cc:1199] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/cbf88a308d6b41fdb6cc2397c95b7612
I20251023 20:15:33.502498 24709 ts_tablet_manager.cc:1950] T cbf88a308d6b41fdb6cc2397c95b7612 P 9348e9ed0255496dbd6e8791382b7a07: Deleting consensus metadata
I20251023 20:15:33.502542 24710 ts_tablet_manager.cc:1929] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:33.502621 24710 log.cc:1199] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/cbf88a308d6b41fdb6cc2397c95b7612
I20251023 20:15:33.502964 24187 catalog_manager.cc:4985] TS 9348e9ed0255496dbd6e8791382b7a07 (127.9.117.194:42639): tablet cbf88a308d6b41fdb6cc2397c95b7612 (table test-table [id=d8a6ff01bf494d25b14e4283a8a67381]) successfully deleted
I20251023 20:15:33.503163 24710 ts_tablet_manager.cc:1950] T cbf88a308d6b41fdb6cc2397c95b7612 P f73de5713f8f45968cc13181138452b0: Deleting consensus metadata
I20251023 20:15:33.503616 24188 catalog_manager.cc:4985] TS f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921): tablet cbf88a308d6b41fdb6cc2397c95b7612 (table test-table [id=d8a6ff01bf494d25b14e4283a8a67381]) successfully deleted
W20251023 20:15:33.507230 24188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:37585 (error 108)
I20251023 20:15:33.507879 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 24403
I20251023 20:15:33.509004 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.507972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:37585 (local address 127.0.0.1:35234)
1023 20:15:33.508688 (+ 716us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.508919 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":680}
W20251023 20:15:33.510980 24188 catalog_manager.cc:4712] TS 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbf88a308d6b41fdb6cc2397c95b7612: Network error: Client connection negotiation failed: client connection to 127.9.117.193:37585: connect: Connection refused (error 111)
I20251023 20:15:33.516589 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 24539
I20251023 20:15:33.525292 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 24171
I20251023 20:15:33.533015 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43961
--webserver_interface=127.9.117.254
--webserver_port=36875
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:33.619807 24712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:33.620033 24712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:33.620074 24712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:33.621767 24712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:33.621833 24712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:33.621855 24712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:33.621868 24712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:33.621881 24712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:33.623709 24712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:43961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=36875
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:33.624135 24712 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:33.624372 24712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:33.632351 24719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:33.632488 24712 server_base.cc:1047] running on GCE node
W20251023 20:15:33.632542 24721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:33.632364 24718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:33.632931 24712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:33.633213 24712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:33.634375 24712 hybrid_clock.cc:648] HybridClock initialized: now 1761250533634347 us; error 44 us; skew 500 ppm
Oct 23 20:15:33 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:33.637081 24712 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:15:33.637738 24712 webserver.cc:492] Webserver started at http://127.9.117.254:36875/ using document root <none> and password file <none>
I20251023 20:15:33.637969 24712 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:33.638028 24712 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:33.639477 24712 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:33.640249 24728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.640513 24712 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:33.640596 24712 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "bd0af698e2ba42349d4a183da8ef3125"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "4a77ecdb4e9236604d46ecc03871d2db"
server_key_iv: "1498cc9261714ab3baa7617944d948a7"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.641010 24712 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:33.670501 24712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:33.672890 24712 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:33.673054 24712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:33.677709 24712 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:43961
I20251023 20:15:33.677757 24780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:43961 every 8 connection(s)
I20251023 20:15:33.678191 24712 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:33.678560 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 24712
I20251023 20:15:33.678964 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:37585
--local_ip_for_outbound_sockets=127.9.117.193
--tserver_master_addrs=127.9.117.254:43961
--webserver_port=37277
--webserver_interface=127.9.117.193
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:15:33.679661 24781 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:15:33.680387 24781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Bootstrap starting.
I20251023 20:15:33.694947 24781 log.cc:826] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:33.697748 24781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:33.698124 24781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Bootstrap complete.
I20251023 20:15:33.701185 24781 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:33.701488 24781 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0af698e2ba42349d4a183da8ef3125, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.701650 24781 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:33.701735 24781 raft_consensus.cc:399] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:15:33.701781 24781 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:15:33.701843 24781 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:33.703148 24781 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:33.703276 24781 leader_election.cc:304] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd0af698e2ba42349d4a183da8ef3125; no voters:
I20251023 20:15:33.703469 24781 leader_election.cc:290] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251023 20:15:33.703547 24785 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:15:33.703719 24781 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:33.703761 24785 raft_consensus.cc:697] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 2 LEADER]: Becoming Leader. State: Replica: bd0af698e2ba42349d4a183da8ef3125, State: Running, Role: LEADER
I20251023 20:15:33.704075 24785 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:33.704414 24786 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bd0af698e2ba42349d4a183da8ef3125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } } }
I20251023 20:15:33.704504 24786 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:33.704986 24794 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:33.705125 24787 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0af698e2ba42349d4a183da8ef3125. Latest consensus state: current_term: 2 leader_uuid: "bd0af698e2ba42349d4a183da8ef3125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } } }
I20251023 20:15:33.705195 24787 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:33.706032 24794 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:33.706243 24794 catalog_manager.cc:1269] Loaded cluster ID: 0fcf776b6f584b83b1ab2c0eff389219
I20251023 20:15:33.706670 24794 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:15:33.706965 24794 catalog_manager.cc:1324] IPKI private key was stored in cleartext, attempting to encrypt it.
I20251023 20:15:33.708541 24794 catalog_manager.cc:1334] IPKI private key successfully encrypted.
I20251023 20:15:33.710760 24794 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:15:33.711056 24794 catalog_manager.cc:6033] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Loaded TSK: 0
I20251023 20:15:33.711280 24794 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251023 20:15:33.787451 24783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:33.787688 24783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:33.787736 24783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:33.789589 24783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:33.789677 24783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:33.789732 24783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:15:33.791754 24783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:37585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=37277
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:33.792078 24783 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:33.792307 24783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:33.796584 24809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:33.796499 24810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:33.796711 24783 server_base.cc:1047] running on GCE node
W20251023 20:15:33.797108 24812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:33.797567 24783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:33.798157 24783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:33.799402 24783 hybrid_clock.cc:648] HybridClock initialized: now 1761250533799368 us; error 104 us; skew 500 ppm
Oct 23 20:15:33 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:33.805997 24783 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:15:33.806965 24783 webserver.cc:492] Webserver started at http://127.9.117.193:37277/ using document root <none> and password file <none>
I20251023 20:15:33.807310 24783 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:33.807402 24783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:33.809929 24783 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:15:33.811224 24819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.811643 24783 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:33.811784 24783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "4315e7383e354eb09243f189ed5f606e"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "c9f9a226b6e9cca12ee161e8ba6daf08"
server_key_iv: "b8b41351d0884f47ce6ab5143a6d63cf"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.812269 24783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:33.831538 24783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:33.833938 24783 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:33.834091 24783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:33.834328 24783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:33.834797 24826 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251023 20:15:33.836084 24783 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251023 20:15:33.836148 24783 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:33.836190 24783 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251023 20:15:33.836849 24783 ts_tablet_manager.cc:616] Registered 1 tablets
I20251023 20:15:33.836903 24783 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:33.836962 24826 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Bootstrap starting.
I20251023 20:15:33.845932 24783 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:37585
I20251023 20:15:33.845961 24933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:37585 every 8 connection(s)
I20251023 20:15:33.846504 24783 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:15:33.850172 24826 log.cc:826] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:33.852979 24826 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:33.853297 24826 tablet_bootstrap.cc:492] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Bootstrap complete.
I20251023 20:15:33.854569 24826 ts_tablet_manager.cc:1403] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.013s
I20251023 20:15:33.856007 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 24783
I20251023 20:15:33.856345 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:42639
--local_ip_for_outbound_sockets=127.9.117.194
--tserver_master_addrs=127.9.117.254:43961
--webserver_port=41595
--webserver_interface=127.9.117.194
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:15:33 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:33.855913 24826 raft_consensus.cc:359] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.856900 24826 raft_consensus.cc:740] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4315e7383e354eb09243f189ed5f606e, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.857096 24826 consensus_queue.cc:260] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } }
I20251023 20:15:33.857455 24826 ts_tablet_manager.cc:1434] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:33.861739 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.847530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:59607 (local address 127.9.117.254:43961)
1023 20:15:33.850607 (+ 3077us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.850623 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.850665 (+ 42us) server_negotiation.cc:408] Connection header received
1023 20:15:33.850844 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.850852 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.850916 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.851097 (+ 181us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:33.852194 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.853606 (+ 1412us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.854526 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.854796 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.857283 (+ 2487us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:33.857294 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:33.857295 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:33.857329 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:33.859747 (+ 2418us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:33.860213 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:33.860218 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:33.860218 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:33.860286 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:33.860484 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:33.860485 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:33.860485 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:33.860918 (+ 433us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:33.861030 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.861385 (+ 355us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.861543 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3005,"thread_start_us":115,"threads_started":1}
I20251023 20:15:33.861759 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.847452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:59607)
1023 20:15:33.847634 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.847677 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.848309 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.851129 (+ 2820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.851134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.851593 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:33.852025 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.852030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.853716 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.853717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.854414 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.854416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.854562 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.855329 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:33.855357 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:33.857131 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:33.859852 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:33.859864 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:33.859867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:33.860121 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:33.860351 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:33.860352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:33.860352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:33.860425 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:33.861005 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:33.861013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:33.861196 (+ 183us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.861308 (+ 112us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.861515 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20251023 20:15:33.862938 24934 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43961
I20251023 20:15:33.863059 24934 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:33.863296 24934 heartbeater.cc:507] Master 127.9.117.254:43961 requested a full tablet report, sending...
I20251023 20:15:33.864198 24745 ts_manager.cc:194] Registered new tserver with Master: 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585)
I20251023 20:15:33.865535 24745 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:59607
I20251023 20:15:33.866451 24934 heartbeater.cc:499] Master 127.9.117.254:43961 was elected leader, sending a full tablet report...
I20251023 20:15:33.873071 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.865055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:37585 (local address 127.0.0.1:35238)
1023 20:15:33.869039 (+ 3984us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:33.869095 (+ 56us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:33.869209 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:33.869406 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:33.869409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:33.869411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:33.870000 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.870003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.871859 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.871860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:33.872650 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:33.872654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.872779 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.872846 (+ 67us) client_negotiation.cc:769] Sending connection context
1023 20:15:33.872907 (+ 61us) client_negotiation.cc:240] Negotiation successful
1023 20:15:33.872966 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3937,"thread_start_us":103,"threads_started":1}
I20251023 20:15:33.873934 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:33.865307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.9.117.193:37585)
1023 20:15:33.869045 (+ 3738us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:33.869061 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:33.869182 (+ 121us) server_negotiation.cc:408] Connection header received
1023 20:15:33.869306 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:33.869308 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:33.869324 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:33.869389 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:33.870096 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.871748 (+ 1652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:33.872787 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:33.873639 (+ 852us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:33.873690 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:33.873750 (+ 60us) server_negotiation.cc:300] Negotiation successful
1023 20:15:33.873815 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3691,"thread_start_us":73,"threads_started":1}
I20251023 20:15:33.874279 24868 tablet_service.cc:1552] Processing DeleteTablet for tablet cbf88a308d6b41fdb6cc2397c95b7612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:33 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:35238
I20251023 20:15:33.874822 24943 tablet_replica.cc:333] stopping tablet replica
I20251023 20:15:33.874933 24943 raft_consensus.cc:2243] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:33.875021 24943 raft_consensus.cc:2272] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.875459 24943 ts_tablet_manager.cc:1916] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.877980 24943 ts_tablet_manager.cc:1929] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:33.878093 24943 log.cc:1199] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/cbf88a308d6b41fdb6cc2397c95b7612
I20251023 20:15:33.878430 24943 ts_tablet_manager.cc:1950] T cbf88a308d6b41fdb6cc2397c95b7612 P 4315e7383e354eb09243f189ed5f606e: Deleting consensus metadata
I20251023 20:15:33.879066 24730 catalog_manager.cc:4985] TS 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585): tablet cbf88a308d6b41fdb6cc2397c95b7612 (table test-table [id=d8a6ff01bf494d25b14e4283a8a67381]) successfully deleted
W20251023 20:15:33.972278 24940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:33.972504 24940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:33.972533 24940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:33.974323 24940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:33.974391 24940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:33.974427 24940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:15:33.976372 24940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:42639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=41595
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:33.976672 24940 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:33.976918 24940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:33.980382 24949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:33.980512 24951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:33.980362 24948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:33.980368 24940 server_base.cc:1047] running on GCE node
I20251023 20:15:33.980996 24940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:33.981273 24940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:33.982419 24940 hybrid_clock.cc:648] HybridClock initialized: now 1761250533982405 us; error 32 us; skew 500 ppm
Oct 23 20:15:33 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:33.985419 24940 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:15:33.986245 24940 webserver.cc:492] Webserver started at http://127.9.117.194:41595/ using document root <none> and password file <none>
I20251023 20:15:33.986490 24940 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:33.986555 24940 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:33.988426 24940 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:15:33.989349 24958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:33.989728 24940 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:33.989838 24940 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "9348e9ed0255496dbd6e8791382b7a07"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "82aa03ac8b230460ab61322eddfb61a5"
server_key_iv: "635c86c987d2dbdc83efc60f16375257"
server_key_version: "encryptionkey@0"
I20251023 20:15:33.990195 24940 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:34.001780 24940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:34.003844 24940 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:34.003998 24940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:34.004236 24940 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:34.004554 24940 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:34.004611 24940 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.004662 24940 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:34.004691 24940 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.011900 24940 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:42639
I20251023 20:15:34.011955 25071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:42639 every 8 connection(s)
I20251023 20:15:34.012328 24940 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:15:34.014339 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 24940
I20251023 20:15:34.014603 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:44921
--local_ip_for_outbound_sockets=127.9.117.195
--tserver_master_addrs=127.9.117.254:43961
--webserver_port=40319
--webserver_interface=127.9.117.195
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:15:34 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250533, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:34.023223 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.013164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:41023 (local address 127.9.117.254:43961)
1023 20:15:34.013237 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.013240 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.013396 (+ 156us) server_negotiation.cc:408] Connection header received
1023 20:15:34.013986 (+ 590us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.013987 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.014000 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.014056 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:34.014983 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.015807 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.016426 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.016617 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.019635 (+ 3018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:34.019643 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:34.019644 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:34.019654 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:34.021772 (+ 2118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:34.022243 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:34.022244 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:34.022245 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:34.022280 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:34.022464 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:34.022465 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:34.022466 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:34.022625 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:34.022708 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.022973 (+ 265us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.023104 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251023 20:15:34.023240 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.013107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:41023)
1023 20:15:34.013284 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.013314 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:34.013902 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:34.014115 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:34.014119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:34.014447 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:34.014880 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.014884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.015917 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.015918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:34.016327 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.016329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.016497 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.017639 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:34.017660 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:34.019518 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:34.021901 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:34.021910 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:34.021913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:34.022158 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:34.022347 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:34.022348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:34.022348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:34.022406 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:34.022689 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:34.022692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:34.022824 (+ 132us) client_negotiation.cc:769] Sending connection context
1023 20:15:34.022931 (+ 107us) client_negotiation.cc:240] Negotiation successful
1023 20:15:34.023057 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20251023 20:15:34.023883 25072 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43961
I20251023 20:15:34.024019 25072 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:34.024296 25072 heartbeater.cc:507] Master 127.9.117.254:43961 requested a full tablet report, sending...
I20251023 20:15:34.024771 24745 ts_manager.cc:194] Registered new tserver with Master: 9348e9ed0255496dbd6e8791382b7a07 (127.9.117.194:42639)
I20251023 20:15:34.025158 24745 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:41023
W20251023 20:15:34.103276 25075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:34.103513 25075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:34.103541 25075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:34.105520 25075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:34.105602 25075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:34.105636 25075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:15:34.107622 25075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:44921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=40319
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:34.107911 25075 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:34.108137 25075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:34.110922 25083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:34.110896 25080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:34.110937 25081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:34.111270 25075 server_base.cc:1047] running on GCE node
I20251023 20:15:34.111456 25075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:34.111716 25075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:34.112876 25075 hybrid_clock.cc:648] HybridClock initialized: now 1761250534112867 us; error 42 us; skew 500 ppm
Oct 23 20:15:34 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:34.115705 25075 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:15:34.116343 25075 webserver.cc:492] Webserver started at http://127.9.117.195:40319/ using document root <none> and password file <none>
I20251023 20:15:34.116557 25075 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:34.116628 25075 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:34.118052 25075 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:34.118793 25090 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.119156 25075 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:34.119240 25075 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "f73de5713f8f45968cc13181138452b0"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "17df9c66e5711382e062c974866fbdc2"
server_key_iv: "db2e7a90e1e22d9d5dacfa1e992daed8"
server_key_version: "encryptionkey@0"
I20251023 20:15:34.119601 25075 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:34.129055 25075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:34.131431 25075 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:34.131561 25075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:34.131839 25075 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:34.132164 25075 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:34.132197 25075 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.132222 25075 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:34.132236 25075 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.138808 25075 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:44921
I20251023 20:15:34.138895 25203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:44921 every 8 connection(s)
I20251023 20:15:34.139235 25075 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:15:34.140144 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 25075
Oct 23 20:15:34 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:34.149075 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.140634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.9.117.254:43961)
1023 20:15:34.140795 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.140797 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.140814 (+ 17us) server_negotiation.cc:408] Connection header received
1023 20:15:34.140902 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.140903 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.140915 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.140985 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:34.141792 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.142346 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.142999 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.143208 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.144496 (+ 1288us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:34.144501 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:34.144501 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:34.144508 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:34.146811 (+ 2303us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:34.147340 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:34.147343 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:34.147344 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:34.147380 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:34.148004 (+ 624us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:34.148006 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:34.148006 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:34.148187 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:34.148394 (+ 207us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.148624 (+ 230us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.148879 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20251023 20:15:34.150832 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.140009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:43743 (local address 127.9.117.254:43961)
1023 20:15:34.140066 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.140070 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.140376 (+ 306us) server_negotiation.cc:408] Connection header received
1023 20:15:34.141060 (+ 684us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.141062 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.141075 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.141141 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:34.142004 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.142494 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.143102 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.143273 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.146548 (+ 3275us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:34.146559 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:34.146560 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:34.146569 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:34.148858 (+ 2289us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:34.149512 (+ 654us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:34.149513 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:34.149514 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:34.149542 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:34.149861 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:34.149862 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:34.149862 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:34.150015 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:34.150068 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.150559 (+ 491us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.150713 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:15:34.151469 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.140013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:43743)
1023 20:15:34.140219 (+ 206us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.140266 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:34.140950 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:34.141212 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:34.141216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:34.141565 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:34.141891 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.141895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.142591 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.142593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:34.143015 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.143017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.143181 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.144036 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:34.144056 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:34.146415 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:34.148993 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:34.149008 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:34.149013 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:34.149412 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:34.149625 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:34.149626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:34.149626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:34.149751 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:34.150128 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:34.150138 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:34.150355 (+ 217us) client_negotiation.cc:769] Sending connection context
1023 20:15:34.150967 (+ 612us) client_negotiation.cc:240] Negotiation successful
1023 20:15:34.151206 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20251023 20:15:34.152128 25204 heartbeater.cc:344] Connected to a master server at 127.9.117.254:43961
I20251023 20:15:34.152257 25204 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:34.152606 25204 heartbeater.cc:507] Master 127.9.117.254:43961 requested a full tablet report, sending...
I20251023 20:15:34.153249 24745 ts_manager.cc:194] Registered new tserver with Master: f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921)
I20251023 20:15:34.153888 24745 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:43743
I20251023 20:15:34.162459 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:15:34.165091 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.162861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.9.117.254:43961)
1023 20:15:34.162912 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.162914 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.163022 (+ 108us) server_negotiation.cc:408] Connection header received
1023 20:15:34.163107 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.163107 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.163136 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.163195 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:15:34.163456 (+ 261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.163968 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.164509 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.164625 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.164685 (+ 60us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:15:34.164870 (+ 185us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:15:34.164944 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.164997 (+ 53us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.165027 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251023 20:15:34.165390 24745 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:34.166922 24745 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:34.169479 24868 tablet_service.cc:1505] Processing CreateTablet for tablet 1232fb4d8918465b95ca8a07a150ecf2 (DEFAULT_TABLE table=test-table [id=f22c572313ce4fc189ee9ddcc23958c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:34.169795 24868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1232fb4d8918465b95ca8a07a150ecf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:34.171624 24826 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Bootstrap starting.
I20251023 20:15:34.172127 24826 tablet_bootstrap.cc:654] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.172443 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.169081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42639 (local address 127.0.0.1:58422)
1023 20:15:34.169313 (+ 232us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.169321 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:34.169367 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:34.169820 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:34.169822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:34.169825 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:34.170050 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.170052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.171204 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.171206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:34.172163 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.172166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.172248 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.172252 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:15:34.172288 (+ 36us) client_negotiation.cc:240] Negotiation successful
1023 20:15:34.172333 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":13,"thread_start_us":62,"threads_started":1}
I20251023 20:15:34.172508 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.169081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:44921 (local address 127.0.0.1:44728)
1023 20:15:34.169203 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.169214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:34.169245 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:34.169716 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:34.169717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:34.169722 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:34.169992 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.169995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.171391 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.171391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:34.172179 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.172181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.172247 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.172252 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:15:34.172288 (+ 36us) client_negotiation.cc:240] Negotiation successful
1023 20:15:34.172328 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20251023 20:15:34.172837 24826 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: No bootstrap required, opened a new log
I20251023 20:15:34.172890 24826 ts_tablet_manager.cc:1403] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:34.173066 24826 raft_consensus.cc:359] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.173152 24826 raft_consensus.cc:385] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.173204 24826 raft_consensus.cc:740] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4315e7383e354eb09243f189ed5f606e, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.173275 24826 consensus_queue.cc:260] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.173393 24826 ts_tablet_manager.cc:1434] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251023 20:15:34.173691 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.169245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.9.117.195:44921)
1023 20:15:34.169513 (+ 268us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.169522 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.169534 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:15:34.169582 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.169585 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.169605 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.170092 (+ 487us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:34.170204 (+ 112us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.171287 (+ 1083us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.172862 (+ 1575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.173485 (+ 623us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.173556 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.173597 (+ 41us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.173639 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20251023 20:15:34.174460 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.169362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.9.117.194:42639)
1023 20:15:34.169605 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.169616 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.169627 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:15:34.169674 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.169677 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.169696 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.170193 (+ 497us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:34.170311 (+ 118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.171106 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.172303 (+ 1197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.174207 (+ 1904us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.174267 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.174307 (+ 40us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.174367 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":144,"threads_started":1}
I20251023 20:15:34.174398 25138 tablet_service.cc:1505] Processing CreateTablet for tablet 1232fb4d8918465b95ca8a07a150ecf2 (DEFAULT_TABLE table=test-table [id=f22c572313ce4fc189ee9ddcc23958c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:34.175046 25138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1232fb4d8918465b95ca8a07a150ecf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:34.175371 25006 tablet_service.cc:1505] Processing CreateTablet for tablet 1232fb4d8918465b95ca8a07a150ecf2 (DEFAULT_TABLE table=test-table [id=f22c572313ce4fc189ee9ddcc23958c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:34.175655 25006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1232fb4d8918465b95ca8a07a150ecf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:34.177695 25214 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Bootstrap starting.
I20251023 20:15:34.178128 25215 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Bootstrap starting.
I20251023 20:15:34.178421 25214 tablet_bootstrap.cc:654] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.178781 25214 log.cc:826] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:34.178813 25215 tablet_bootstrap.cc:654] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.179131 25215 log.cc:826] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:34.179678 25214 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: No bootstrap required, opened a new log
I20251023 20:15:34.179735 25214 ts_tablet_manager.cc:1403] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.179914 25215 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: No bootstrap required, opened a new log
I20251023 20:15:34.179984 25215 ts_tablet_manager.cc:1403] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.181313 25214 raft_consensus.cc:359] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.181495 25214 raft_consensus.cc:385] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.181543 25214 raft_consensus.cc:740] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73de5713f8f45968cc13181138452b0, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.181595 25215 raft_consensus.cc:359] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.181680 25214 consensus_queue.cc:260] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.181735 25215 raft_consensus.cc:385] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.181768 25215 raft_consensus.cc:740] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9348e9ed0255496dbd6e8791382b7a07, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.181900 25215 consensus_queue.cc:260] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.182050 25204 heartbeater.cc:499] Master 127.9.117.254:43961 was elected leader, sending a full tablet report...
I20251023 20:15:34.182181 25214 ts_tablet_manager.cc:1434] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:15:34.182185 25072 heartbeater.cc:499] Master 127.9.117.254:43961 was elected leader, sending a full tablet report...
I20251023 20:15:34.182466 25215 ts_tablet_manager.cc:1434] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20251023 20:15:34.262894 25073 tablet.cc:2378] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:34.318168 25218 raft_consensus.cc:493] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.318320 25218 raft_consensus.cc:515] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.318744 25218 leader_election.cc:290] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9348e9ed0255496dbd6e8791382b7a07 (127.9.117.194:42639), 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585)
I20251023 20:15:34.321240 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.318936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42639 (local address 127.9.117.195:46001)
1023 20:15:34.319003 (+ 67us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.319013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:34.319065 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:34.319223 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:34.319224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:34.319225 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:34.319461 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.319463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.320308 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.320309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:34.321060 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.321062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.321123 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.321128 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:15:34.321152 (+ 24us) client_negotiation.cc:240] Negotiation successful
1023 20:15:34.321184 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251023 20:15:34.321592 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.318948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:37585 (local address 127.9.117.195:51539)
1023 20:15:34.319120 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.319130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:34.319198 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:34.319357 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:34.319358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:34.319360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:34.319616 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.319617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.320520 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.320521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:34.321326 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:34.321329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.321436 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.321439 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:15:34.321474 (+ 35us) client_negotiation.cc:240] Negotiation successful
1023 20:15:34.321510 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
I20251023 20:15:34.321769 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.319019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:46001 (local address 127.9.117.194:42639)
1023 20:15:34.319089 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.319091 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.319114 (+ 23us) server_negotiation.cc:408] Connection header received
1023 20:15:34.319149 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.319150 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.319161 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.319209 (+ 48us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:34.319562 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.320234 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.321142 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.321632 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.321643 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.321668 (+ 25us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.321706 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251023 20:15:34.322048 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.319158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:51539 (local address 127.9.117.193:37585)
1023 20:15:34.319236 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.319239 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.319249 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:15:34.319280 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.319282 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.319293 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.319345 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:34.319706 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.320418 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.321442 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.321914 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.321921 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.321942 (+ 21us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.321974 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251023 20:15:34.322113 25026 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1232fb4d8918465b95ca8a07a150ecf2" candidate_uuid: "f73de5713f8f45968cc13181138452b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9348e9ed0255496dbd6e8791382b7a07" is_pre_election: true
I20251023 20:15:34.322274 25026 raft_consensus.cc:2468] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f73de5713f8f45968cc13181138452b0 in term 0.
I20251023 20:15:34.322355 24888 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1232fb4d8918465b95ca8a07a150ecf2" candidate_uuid: "f73de5713f8f45968cc13181138452b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4315e7383e354eb09243f189ed5f606e" is_pre_election: true
I20251023 20:15:34.322496 25091 leader_election.cc:304] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9348e9ed0255496dbd6e8791382b7a07, f73de5713f8f45968cc13181138452b0; no voters:
I20251023 20:15:34.322536 24888 raft_consensus.cc:2468] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f73de5713f8f45968cc13181138452b0 in term 0.
I20251023 20:15:34.322630 25218 raft_consensus.cc:2804] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.322706 25218 raft_consensus.cc:493] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.322726 25218 raft_consensus.cc:3060] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.323452 25218 raft_consensus.cc:515] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.323580 25218 leader_election.cc:290] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [CANDIDATE]: Term 1 election: Requested vote from peers 9348e9ed0255496dbd6e8791382b7a07 (127.9.117.194:42639), 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585)
I20251023 20:15:34.323803 24888 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1232fb4d8918465b95ca8a07a150ecf2" candidate_uuid: "f73de5713f8f45968cc13181138452b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4315e7383e354eb09243f189ed5f606e"
I20251023 20:15:34.323807 25026 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1232fb4d8918465b95ca8a07a150ecf2" candidate_uuid: "f73de5713f8f45968cc13181138452b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9348e9ed0255496dbd6e8791382b7a07"
I20251023 20:15:34.323889 24888 raft_consensus.cc:3060] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.323895 25026 raft_consensus.cc:3060] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.324661 25026 raft_consensus.cc:2468] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f73de5713f8f45968cc13181138452b0 in term 1.
I20251023 20:15:34.324668 24888 raft_consensus.cc:2468] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f73de5713f8f45968cc13181138452b0 in term 1.
I20251023 20:15:34.324842 25091 leader_election.cc:304] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9348e9ed0255496dbd6e8791382b7a07, f73de5713f8f45968cc13181138452b0; no voters:
I20251023 20:15:34.325105 25218 raft_consensus.cc:2804] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:34.325277 25218 raft_consensus.cc:697] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 1 LEADER]: Becoming Leader. State: Replica: f73de5713f8f45968cc13181138452b0, State: Running, Role: LEADER
I20251023 20:15:34.325384 25218 consensus_queue.cc:237] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.326344 24744 catalog_manager.cc:5649] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f73de5713f8f45968cc13181138452b0 (127.9.117.195). New cstate: current_term: 1 leader_uuid: "f73de5713f8f45968cc13181138452b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } health_report { overall_health: UNKNOWN } } }
W20251023 20:15:34.351092 24935 tablet.cc:2378] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:34.361007 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.358845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.9.117.195:44921)
1023 20:15:34.358920 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:34.358922 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:34.358930 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:15:34.358976 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:34.358977 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:34.358996 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:34.359047 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:15:34.359259 (+ 212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.359749 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:34.360336 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:34.360476 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:34.360519 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:15:34.360654 (+ 135us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:15:34.360722 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:34.360886 (+ 164us) server_negotiation.cc:300] Negotiation successful
1023 20:15:34.360914 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251023 20:15:34.362879 25026 raft_consensus.cc:1275] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Refusing update from remote peer f73de5713f8f45968cc13181138452b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:34.362874 24888 raft_consensus.cc:1275] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Refusing update from remote peer f73de5713f8f45968cc13181138452b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:34.363250 25221 consensus_queue.cc:1048] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.363402 25218 consensus_queue.cc:1048] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.365970 25226 mvcc.cc:204] Tried to move back new op lower bound from 7214082188747726848 to 7214082188597477376. Current Snapshot: MvccSnapshot[applied={T|T < 7214082188747726848}]
I20251023 20:15:34.366001 25225 mvcc.cc:204] Tried to move back new op lower bound from 7214082188747726848 to 7214082188597477376. Current Snapshot: MvccSnapshot[applied={T|T < 7214082188747726848}]
I20251023 20:15:34.368352 24745 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51600:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:15:34.368479 24745 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51600:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:15:34.369305 24745 catalog_manager.cc:5936] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Sending DeleteTablet for 3 replicas of tablet 1232fb4d8918465b95ca8a07a150ecf2
I20251023 20:15:34.369717 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 24783
I20251023 20:15:34.369768 24868 tablet_service.cc:1552] Processing DeleteTablet for tablet 1232fb4d8918465b95ca8a07a150ecf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:34 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:35238
I20251023 20:15:34.369791 25006 tablet_service.cc:1552] Processing DeleteTablet for tablet 1232fb4d8918465b95ca8a07a150ecf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:34 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:58422
I20251023 20:15:34.369882 24943 tablet_replica.cc:333] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: stopping tablet replica
I20251023 20:15:34.369935 24943 raft_consensus.cc:2243] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:34.369985 24943 raft_consensus.cc:2272] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:34.370285 25138 tablet_service.cc:1552] Processing DeleteTablet for tablet 1232fb4d8918465b95ca8a07a150ecf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:34 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:44728
I20251023 20:15:34.370537 25235 tablet_replica.cc:333] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: stopping tablet replica
I20251023 20:15:34.370635 25235 raft_consensus.cc:2243] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:34.370781 25235 raft_consensus.cc:2272] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:34.371287 25235 ts_tablet_manager.cc:1916] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:34.373795 25235 ts_tablet_manager.cc:1929] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:34.373890 25235 log.cc:1199] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/1232fb4d8918465b95ca8a07a150ecf2
I20251023 20:15:34.374150 25235 ts_tablet_manager.cc:1950] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Deleting consensus metadata
I20251023 20:15:34.374616 24730 catalog_manager.cc:4985] TS f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921): tablet 1232fb4d8918465b95ca8a07a150ecf2 (table test-table [id=f22c572313ce4fc189ee9ddcc23958c6]) successfully deleted
W20251023 20:15:34.378664 24730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:37585 (error 108)
I20251023 20:15:34.378955 25234 tablet_replica.cc:333] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: stopping tablet replica
I20251023 20:15:34.379002 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 24940
I20251023 20:15:34.379088 25234 raft_consensus.cc:2243] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:34.379242 25234 raft_consensus.cc:2272] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:34.382294 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.382105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:37585 (local address 127.0.0.1:35244)
1023 20:15:34.382181 (+ 76us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.382235 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20251023 20:15:34.382511 24730 catalog_manager.cc:4712] TS 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1232fb4d8918465b95ca8a07a150ecf2: Network error: Client connection negotiation failed: client connection to 127.9.117.193:37585: connect: Connection refused (error 111)
I20251023 20:15:34.385373 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.385220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42639 (local address 127.0.0.1:58438)
1023 20:15:34.385286 (+ 66us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.385317 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.194:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20251023 20:15:34.385447 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 25075
I20251023 20:15:34.390813 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 24712
I20251023 20:15:34.397985 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43961
--webserver_interface=127.9.117.254
--webserver_port=36875
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:34.486275 25237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:34.486516 25237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:34.486545 25237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:34.488359 25237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:34.488436 25237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:34.488451 25237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:34.488466 25237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:34.488477 25237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:34.490520 25237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:43961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=36875
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:34.490852 25237 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:34.491093 25237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:34.493906 25243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:34.494004 25242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:34.494135 25237 server_base.cc:1047] running on GCE node
W20251023 20:15:34.494004 25245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:34.494544 25237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:34.494858 25237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:34.496031 25237 hybrid_clock.cc:648] HybridClock initialized: now 1761250534496011 us; error 39 us; skew 500 ppm
Oct 23 20:15:34 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:34.499032 25237 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:15:34.499711 25237 webserver.cc:492] Webserver started at http://127.9.117.254:36875/ using document root <none> and password file <none>
I20251023 20:15:34.499913 25237 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:34.499958 25237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:34.501345 25237 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:34.502049 25252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.502300 25237 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:34.502395 25237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "bd0af698e2ba42349d4a183da8ef3125"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "4a77ecdb4e9236604d46ecc03871d2db"
server_key_iv: "1498cc9261714ab3baa7617944d948a7"
server_key_version: "encryptionkey@0"
I20251023 20:15:34.502714 25237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:34.520117 25237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:34.522215 25237 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:34.522358 25237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:34.526777 25237 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:43961
I20251023 20:15:34.526880 25304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:43961 every 8 connection(s)
I20251023 20:15:34.527206 25237 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:34.528429 25305 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:15:34.529117 25305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Bootstrap starting.
I20251023 20:15:34.533239 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 25237
I20251023 20:15:34.533540 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:37585
--local_ip_for_outbound_sockets=127.9.117.193
--tserver_master_addrs=127.9.117.254:43961
--webserver_port=37277
--webserver_interface=127.9.117.193
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:15:34.539148 25305 log.cc:826] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:34.541551 25305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:34.541857 25305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125: Bootstrap complete.
I20251023 20:15:34.544032 25305 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:34.544251 25305 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0af698e2ba42349d4a183da8ef3125, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.544406 25305 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:34.544492 25305 raft_consensus.cc:399] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:15:34.544543 25305 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:15:34.544597 25305 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 2 FOLLOWER]: Advancing to term 3
I20251023 20:15:34.545584 25305 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:34.545696 25305 leader_election.cc:304] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd0af698e2ba42349d4a183da8ef3125; no voters:
I20251023 20:15:34.545869 25305 leader_election.cc:290] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [CANDIDATE]: Term 3 election: Requested vote from peers
I20251023 20:15:34.545949 25309 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 3 FOLLOWER]: Leader election won for term 3
I20251023 20:15:34.546119 25309 raft_consensus.cc:697] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [term 3 LEADER]: Becoming Leader. State: Replica: bd0af698e2ba42349d4a183da8ef3125, State: Running, Role: LEADER
I20251023 20:15:34.546154 25305 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:34.546211 25309 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } }
I20251023 20:15:34.546521 25309 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0af698e2ba42349d4a183da8ef3125. Latest consensus state: current_term: 3 leader_uuid: "bd0af698e2ba42349d4a183da8ef3125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } } }
I20251023 20:15:34.546589 25309 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:34.546556 25310 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bd0af698e2ba42349d4a183da8ef3125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0af698e2ba42349d4a183da8ef3125" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 43961 } } }
I20251023 20:15:34.546828 25310 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0af698e2ba42349d4a183da8ef3125 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:34.546779 25314 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:34.547804 25314 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:34.548002 25314 catalog_manager.cc:1269] Loaded cluster ID: 0fcf776b6f584b83b1ab2c0eff389219
I20251023 20:15:34.548269 25314 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
F20251023 20:15:34.548588 25314 catalog_manager.cc:1476] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd2d299edcd google::LogMessage::Fail() at ??:0
@ 0x7fd2d29a2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd2d299e7cc google::LogMessage::Flush() at ??:0
@ 0x7fd2d299ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd2d51ecbd6 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd2d51f7334 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd2d3aa02b5 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd2d3a964e7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd2d3dc56db start_thread at ??:0
@ 0x7fd2d1ef371f clone at ??:0
W20251023 20:15:34.636224 25307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:34.636464 25307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:34.636511 25307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:34.638274 25307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:34.638348 25307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:34.638401 25307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:15:34.640360 25307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:37585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=37277
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:34.640682 25307 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:34.640936 25307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:34.643947 25344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:34.643950 25341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:34.644176 25307 server_base.cc:1047] running on GCE node
W20251023 20:15:34.643970 25342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:34.644464 25307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:34.644779 25307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:34.645972 25307 hybrid_clock.cc:648] HybridClock initialized: now 1761250534645948 us; error 43 us; skew 500 ppm
Oct 23 20:15:34 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:34.649546 25307 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:15:34.650197 25307 webserver.cc:492] Webserver started at http://127.9.117.193:37277/ using document root <none> and password file <none>
I20251023 20:15:34.650413 25307 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:34.650451 25307 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:34.651762 25307 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:34.652418 25351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.652621 25307 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.652722 25307 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "4315e7383e354eb09243f189ed5f606e"
format_stamp: "Formatted at 2025-10-23 20:15:32 on dist-test-slave-x62p"
server_key: "c9f9a226b6e9cca12ee161e8ba6daf08"
server_key_iv: "b8b41351d0884f47ce6ab5143a6d63cf"
server_key_version: "encryptionkey@0"
I20251023 20:15:34.653154 25307 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:34.668610 25307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:34.671401 25307 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:34.671578 25307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:34.671900 25307 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:34.672487 25358 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251023 20:15:34.673544 25307 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251023 20:15:34.673612 25307 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:34.673651 25307 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251023 20:15:34.674539 25307 ts_tablet_manager.cc:616] Registered 1 tablets
I20251023 20:15:34.674589 25307 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:34.674645 25358 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Bootstrap starting.
I20251023 20:15:34.683357 25307 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:37585
I20251023 20:15:34.683804 25307 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:15:34.684926 25465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:37585 every 8 connection(s)
I20251023 20:15:34.687047 25358 log.cc:826] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:34.688942 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 25307
I20251023 20:15:34.689242 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:42639
--local_ip_for_outbound_sockets=127.9.117.194
--tserver_master_addrs=127.9.117.254:43961
--webserver_port=41595
--webserver_interface=127.9.117.194
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:15:34.689261 25358 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:34.689608 25358 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Bootstrap complete.
I20251023 20:15:34.690620 25358 ts_tablet_manager.cc:1403] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20251023 20:15:34.691954 25358 raft_consensus.cc:359] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.692183 25358 raft_consensus.cc:740] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4315e7383e354eb09243f189ed5f606e, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.692338 25358 consensus_queue.cc:260] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.692651 25358 ts_tablet_manager.cc:1434] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20251023 20:15:34.760272 25468 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:15:34.684531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:37527)
1023 20:15:34.684704 (+ 173us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.684758 (+ 54us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:34.685740 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:34.760084 (+ 74344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
W20251023 20:15:34.760581 25466 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:43961 (0 consecutive failures): Network error: Failed to ping master at 127.9.117.254:43961: Client connection negotiation failed: client connection to 127.9.117.254:43961: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:34.761067 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.760903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:35957)
1023 20:15:34.760976 (+ 73us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.761024 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20251023 20:15:34.761466 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.761346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:35657)
1023 20:15:34.761400 (+ 54us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.761432 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251023 20:15:34.761690 25466 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251023 20:15:34.775138 25470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:34.775350 25470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:34.775380 25470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:34.777118 25470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:34.777179 25470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:34.777211 25470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:15:34.778982 25470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:42639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=41595
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:34.779248 25470 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:34.779451 25470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:34.782495 25479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:34.782485 25476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:34.782486 25477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:34.782541 25470 server_base.cc:1047] running on GCE node
I20251023 20:15:34.782855 25470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:34.783088 25470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:34.784231 25470 hybrid_clock.cc:648] HybridClock initialized: now 1761250534784213 us; error 30 us; skew 500 ppm
Oct 23 20:15:34 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:34.786890 25470 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:15:34.787444 25470 webserver.cc:492] Webserver started at http://127.9.117.194:41595/ using document root <none> and password file <none>
I20251023 20:15:34.787645 25470 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:34.787689 25470 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:34.789148 25470 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:34.789850 25486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.790071 25470 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:34.790148 25470 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "9348e9ed0255496dbd6e8791382b7a07"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "82aa03ac8b230460ab61322eddfb61a5"
server_key_iv: "635c86c987d2dbdc83efc60f16375257"
server_key_version: "encryptionkey@0"
I20251023 20:15:34.790459 25470 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:34.798717 25470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:34.800837 25470 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:34.801005 25470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:34.801259 25470 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:34.801767 25493 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251023 20:15:34.802776 25470 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251023 20:15:34.802838 25470 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:34.802889 25470 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251023 20:15:34.803536 25470 ts_tablet_manager.cc:616] Registered 1 tablets
I20251023 20:15:34.803580 25470 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:34.803676 25493 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Bootstrap starting.
I20251023 20:15:34.810936 25470 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:42639
I20251023 20:15:34.811015 25600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:42639 every 8 connection(s)
I20251023 20:15:34.811332 25470 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:15:34.812575 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.812085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:41123)
1023 20:15:34.812272 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.812372 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
W20251023 20:15:34.812865 25601 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:43961 (0 consecutive failures): Network error: Failed to ping master at 127.9.117.254:43961: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
I20251023 20:15:34.813249 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.813113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:56255)
1023 20:15:34.813168 (+ 55us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.813209 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20251023 20:15:34.813647 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.813527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:39099)
1023 20:15:34.813581 (+ 54us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.813615 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251023 20:15:34.813782 25601 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251023 20:15:34.815152 25493 log.cc:826] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:34.815922 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 25470
I20251023 20:15:34.816236 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:44921
--local_ip_for_outbound_sockets=127.9.117.195
--tserver_master_addrs=127.9.117.254:43961
--webserver_port=40319
--webserver_interface=127.9.117.195
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:15:34.818099 25493 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:34.818370 25493 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Bootstrap complete.
I20251023 20:15:34.819428 25493 ts_tablet_manager.cc:1403] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20251023 20:15:34.820533 25493 raft_consensus.cc:359] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.820824 25493 raft_consensus.cc:740] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9348e9ed0255496dbd6e8791382b7a07, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.820938 25493 consensus_queue.cc:260] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:34.821197 25493 ts_tablet_manager.cc:1434] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:34.821521 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.821419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:53075)
1023 20:15:34.821459 (+ 40us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.821494 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20251023 20:15:34.821844 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.821758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:40511)
1023 20:15:34.821802 (+ 44us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.821824 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20251023 20:15:34.904765 25605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:34.905004 25605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:34.905046 25605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:34.906726 25605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:34.906793 25605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:34.906854 25605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:15:34.908864 25605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:44921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=40319
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:34.909164 25605 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:34.909371 25605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:34.913165 25612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:34.913218 25611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:34.913165 25614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:34.913228 25605 server_base.cc:1047] running on GCE node
I20251023 20:15:34.913580 25605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:34.913817 25605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:34.914960 25605 hybrid_clock.cc:648] HybridClock initialized: now 1761250534914931 us; error 44 us; skew 500 ppm
Oct 23 20:15:34 dist-test-slave-x62p krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:34.917814 25605 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:15:34.918438 25605 webserver.cc:492] Webserver started at http://127.9.117.195:40319/ using document root <none> and password file <none>
I20251023 20:15:34.918658 25605 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:34.918701 25605 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:34.920296 25605 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:15:34.921032 25621 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.921305 25605 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:34.921411 25605 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "f73de5713f8f45968cc13181138452b0"
format_stamp: "Formatted at 2025-10-23 20:15:33 on dist-test-slave-x62p"
server_key: "17df9c66e5711382e062c974866fbdc2"
server_key_iv: "db2e7a90e1e22d9d5dacfa1e992daed8"
server_key_version: "encryptionkey@0"
I20251023 20:15:34.921794 25605 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:34.930160 25605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:34.932255 25605 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:34.932365 25605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:34.932586 25605 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:34.932960 25605 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:34.932997 25605 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.933020 25605 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:34.933032 25605 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:34.939276 25605 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:44921
I20251023 20:15:34.939395 25734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:44921 every 8 connection(s)
I20251023 20:15:34.939685 25605 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:15:34.940882 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.940412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:41519)
1023 20:15:34.940611 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.940699 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
W20251023 20:15:34.941114 25735 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:43961 (0 consecutive failures): Network error: Failed to ping master at 127.9.117.254:43961: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
I20251023 20:15:34.941392 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.941301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:47665)
1023 20:15:34.941345 (+ 44us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.941373 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20251023 20:15:34.941643 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:34.941570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:34995)
1023 20:15:34.941606 (+ 36us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:34.941626 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20251023 20:15:34.941738 25735 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251023 20:15:34.942536 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 25605
I20251023 20:15:35.762404 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:35.762130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:52009)
1023 20:15:35.762301 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:35.762348 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20251023 20:15:35.822571 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:35.822321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:36217)
1023 20:15:35.822476 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:35.822518 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
I20251023 20:15:35.942312 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:35.942046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:45079)
1023 20:15:35.942203 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:35.942245 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20251023 20:15:35.956543 25742 raft_consensus.cc:493] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.956686 25742 raft_consensus.cc:515] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:35.957049 25742 leader_election.cc:290] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921), 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585)
Oct 23 20:15:35 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.193@KRBTEST.COM
Oct 23 20:15:35 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.195@KRBTEST.COM
I20251023 20:15:35.971959 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:35.957526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:36481 (local address 127.9.117.193:37585)
1023 20:15:35.957699 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:35.957719 (+ 20us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:35.957729 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:15:35.957815 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:35.957817 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:35.957840 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:35.957935 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:35.958924 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:35.960187 (+ 1263us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:35.960942 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:35.961177 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:35.967848 (+ 6671us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:35.967863 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:35.967864 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:35.967927 (+ 63us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:35.970028 (+ 2101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:35.970586 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:35.970590 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:35.970590 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:35.970679 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:35.970970 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:35.970971 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:35.970971 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:35.971266 (+ 295us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:35.971345 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:35.971687 (+ 342us) server_negotiation.cc:300] Negotiation successful
1023 20:15:35.971835 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20251023 20:15:35.971972 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:35.957380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:53115 (local address 127.9.117.195:44921)
1023 20:15:35.957567 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:35.957584 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:35.957610 (+ 26us) server_negotiation.cc:408] Connection header received
1023 20:15:35.958282 (+ 672us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:35.958285 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:35.958323 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:35.958418 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:35.958922 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:35.960393 (+ 1471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:35.961057 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:35.961287 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:35.968182 (+ 6895us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:35.968195 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:35.968196 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:35.968234 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:35.970309 (+ 2075us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:35.970753 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:35.970758 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:35.970758 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:35.970838 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:35.971103 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:35.971104 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:35.971104 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:35.971342 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:35.971441 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:35.971724 (+ 283us) server_negotiation.cc:300] Negotiation successful
1023 20:15:35.971855 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20251023 20:15:35.971980 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:35.957366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:37585 (local address 127.9.117.194:36481)
1023 20:15:35.957484 (+ 118us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:35.957495 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:35.957606 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:35.957998 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:35.958003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:35.958357 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:35.958777 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:35.958782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:35.960306 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:35.960307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:35.960852 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:35.960855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:35.960992 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:35.965605 (+ 4613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:35.965663 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:35.967677 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:35.970151 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:35.970164 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:35.970167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:35.970501 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:35.970772 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:35.970773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:35.970773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:35.970869 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:35.971352 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:35.971359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:35.971505 (+ 146us) client_negotiation.cc:769] Sending connection context
1023 20:15:35.971609 (+ 104us) client_negotiation.cc:240] Negotiation successful
1023 20:15:35.971803 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20251023 20:15:35.972558 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:35.957252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:44921 (local address 127.9.117.194:53115)
1023 20:15:35.957305 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:35.957328 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:35.957676 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:35.958433 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:35.958434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:35.958579 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:35.958812 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:35.958814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:35.960474 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:35.960475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:35.960986 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:35.960988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:35.961048 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:35.965618 (+ 4570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:35.965663 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:35.967998 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:35.970400 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:35.970401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:35.970402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:35.970611 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:35.970970 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:35.970971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:35.970971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:35.971009 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:35.971423 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:35.971423 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:35.971505 (+ 82us) client_negotiation.cc:769] Sending connection context
1023 20:15:35.971655 (+ 150us) client_negotiation.cc:240] Negotiation successful
1023 20:15:35.971890 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":116}
I20251023 20:15:35.972668 25420 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1232fb4d8918465b95ca8a07a150ecf2" candidate_uuid: "9348e9ed0255496dbd6e8791382b7a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4315e7383e354eb09243f189ed5f606e" is_pre_election: true
I20251023 20:15:35.972883 25420 raft_consensus.cc:2468] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9348e9ed0255496dbd6e8791382b7a07 in term 1.
I20251023 20:15:35.973183 25488 leader_election.cc:304] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4315e7383e354eb09243f189ed5f606e, 9348e9ed0255496dbd6e8791382b7a07; no voters:
W20251023 20:15:35.973337 25488 leader_election.cc:343] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921): Not found: Tablet not found: 1232fb4d8918465b95ca8a07a150ecf2
I20251023 20:15:35.973378 25742 raft_consensus.cc:2804] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 20:15:35.973570 25742 raft_consensus.cc:493] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.973613 25742 raft_consensus.cc:3060] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:35.974915 25742 raft_consensus.cc:515] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:35.975073 25742 leader_election.cc:290] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 2 election: Requested vote from peers f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921), 4315e7383e354eb09243f189ed5f606e (127.9.117.193:37585)
I20251023 20:15:35.975342 25420 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1232fb4d8918465b95ca8a07a150ecf2" candidate_uuid: "9348e9ed0255496dbd6e8791382b7a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4315e7383e354eb09243f189ed5f606e"
I20251023 20:15:35.975451 25420 raft_consensus.cc:3060] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 1 FOLLOWER]: Advancing to term 2
W20251023 20:15:35.975505 25488 leader_election.cc:343] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921): Not found: Tablet not found: 1232fb4d8918465b95ca8a07a150ecf2
I20251023 20:15:35.976897 25420 raft_consensus.cc:2468] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9348e9ed0255496dbd6e8791382b7a07 in term 2.
I20251023 20:15:35.977833 25488 leader_election.cc:304] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4315e7383e354eb09243f189ed5f606e, 9348e9ed0255496dbd6e8791382b7a07; no voters: f73de5713f8f45968cc13181138452b0
I20251023 20:15:35.978190 25742 raft_consensus.cc:2804] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:15:35.980278 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:35.980158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:42207)
1023 20:15:35.980209 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:35.980243 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20251023 20:15:35.980757 25742 raft_consensus.cc:697] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [term 2 LEADER]: Becoming Leader. State: Replica: 9348e9ed0255496dbd6e8791382b7a07, State: Running, Role: LEADER
I20251023 20:15:35.981283 25742 consensus_queue.cc:237] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:36.378374 25420 raft_consensus.cc:1275] T 1232fb4d8918465b95ca8a07a150ecf2 P 4315e7383e354eb09243f189ed5f606e [term 2 FOLLOWER]: Refusing update from remote peer 9348e9ed0255496dbd6e8791382b7a07: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251023 20:15:36.378777 25742 consensus_queue.cc:1048] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251023 20:15:36.379928 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:36.379667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:59595)
1023 20:15:36.379788 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:36.379848 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20251023 20:15:36.380910 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:36.380767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:44339)
1023 20:15:36.380818 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:36.380865 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20251023 20:15:36.381067 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:36.380914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:54641)
1023 20:15:36.380955 (+ 41us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:36.380990 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20251023 20:15:36.381079 25488 consensus_peers.cc:597] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07 -> Peer f73de5713f8f45968cc13181138452b0 (127.9.117.195:44921): Couldn't send request to peer f73de5713f8f45968cc13181138452b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1232fb4d8918465b95ca8a07a150ecf2. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:36.943195 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:36.942916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:53853)
1023 20:15:36.943094 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:36.943141 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20251023 20:15:36.968077 25755 ts_tablet_manager.cc:933] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Initiating tablet copy from peer 9348e9ed0255496dbd6e8791382b7a07 (127.9.117.194:42639)
I20251023 20:15:36.968513 25755 tablet_copy_client.cc:323] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: tablet copy: Beginning tablet copy session from remote peer at address 127.9.117.194:42639
Oct 23 20:15:36 dist-test-slave-x62p krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250534, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.194@KRBTEST.COM
I20251023 20:15:36.982533 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:36.969361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:35183 (local address 127.9.117.194:42639)
1023 20:15:36.969581 (+ 220us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:36.969613 (+ 32us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:36.969640 (+ 27us) server_negotiation.cc:408] Connection header received
1023 20:15:36.969703 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:36.969708 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:36.969860 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:36.970004 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:36.971105 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:36.972119 (+ 1014us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:36.972974 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:36.973219 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:36.976522 (+ 3303us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:36.976545 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:36.976545 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:36.976562 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:36.980030 (+ 3468us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:36.980944 (+ 914us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:36.980949 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:36.980950 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:36.981033 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:36.981344 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:36.981344 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:36.981345 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:36.981617 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:36.981894 (+ 277us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:36.982149 (+ 255us) server_negotiation.cc:300] Negotiation successful
1023 20:15:36.982341 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20251023 20:15:36.982582 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:36.969070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42639 (local address 127.9.117.195:35183)
1023 20:15:36.969192 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:36.969227 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:36.969361 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:36.970060 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:36.970074 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:36.970658 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:36.970986 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:36.970989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:36.972264 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:36.972265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:36.972870 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:36.972873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:36.972975 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:36.973869 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:36.973876 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:36.976330 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:36.980223 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:36.980236 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:36.980245 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:36.980795 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:36.981189 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:36.981190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:36.981190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:36.981245 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:36.981747 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:36.981767 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:36.981914 (+ 147us) client_negotiation.cc:769] Sending connection context
1023 20:15:36.982083 (+ 169us) client_negotiation.cc:240] Negotiation successful
1023 20:15:36.982343 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20251023 20:15:36.983467 25575 tablet_copy_service.cc:140] P 9348e9ed0255496dbd6e8791382b7a07: Received BeginTabletCopySession request for tablet 1232fb4d8918465b95ca8a07a150ecf2 from peer f73de5713f8f45968cc13181138452b0 ({username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:35183)
I20251023 20:15:36.983582 25575 tablet_copy_service.cc:161] P 9348e9ed0255496dbd6e8791382b7a07: Beginning new tablet copy session on tablet 1232fb4d8918465b95ca8a07a150ecf2 from peer f73de5713f8f45968cc13181138452b0 at {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:35183: session id = f73de5713f8f45968cc13181138452b0-1232fb4d8918465b95ca8a07a150ecf2
I20251023 20:15:36.984234 25575 tablet_copy_source_session.cc:215] T 1232fb4d8918465b95ca8a07a150ecf2 P 9348e9ed0255496dbd6e8791382b7a07: Tablet Copy: opened 0 blocks and 1 log segments
I20251023 20:15:36.985324 25755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1232fb4d8918465b95ca8a07a150ecf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.987903 25755 tablet_copy_client.cc:806] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: tablet copy: Starting download of 0 data blocks...
I20251023 20:15:36.988071 25755 tablet_copy_client.cc:670] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: tablet copy: Starting download of 1 WAL segments...
I20251023 20:15:36.989404 25755 tablet_copy_client.cc:538] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251023 20:15:36.990348 25755 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Bootstrap starting.
I20251023 20:15:36.991992 25755 log.cc:826] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:36.994391 25755 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:36.994656 25755 tablet_bootstrap.cc:492] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Bootstrap complete.
I20251023 20:15:36.994786 25755 ts_tablet_manager.cc:1403] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251023 20:15:36.995679 25755 raft_consensus.cc:359] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:36.995824 25755 raft_consensus.cc:740] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73de5713f8f45968cc13181138452b0, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.995929 25755 consensus_queue.cc:260] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73de5713f8f45968cc13181138452b0" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 44921 } } peers { permanent_uuid: "9348e9ed0255496dbd6e8791382b7a07" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42639 } } peers { permanent_uuid: "4315e7383e354eb09243f189ed5f606e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 37585 } }
I20251023 20:15:36.996188 25755 ts_tablet_manager.cc:1434] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20251023 20:15:36.996618 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:36.996498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:45265)
1023 20:15:36.996540 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:36.996577 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20251023 20:15:36.996644 25575 tablet_copy_service.cc:342] P 9348e9ed0255496dbd6e8791382b7a07: Request end of tablet copy session f73de5713f8f45968cc13181138452b0-1232fb4d8918465b95ca8a07a150ecf2 received from {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:35183
I20251023 20:15:36.996745 25575 tablet_copy_service.cc:434] P 9348e9ed0255496dbd6e8791382b7a07: ending tablet copy session f73de5713f8f45968cc13181138452b0-1232fb4d8918465b95ca8a07a150ecf2 on tablet 1232fb4d8918465b95ca8a07a150ecf2 with peer f73de5713f8f45968cc13181138452b0
I20251023 20:15:37.381765 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:37.381433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:57105)
1023 20:15:37.381649 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:37.381697 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20251023 20:15:37.381830 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:37.381616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:38149)
1023 20:15:37.381749 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:37.381782 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20251023 20:15:37.413538 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:37.413386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:40107)
1023 20:15:37.413447 (+ 61us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:37.413491 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251023 20:15:37.413843 25689 raft_consensus.cc:1217] T 1232fb4d8918465b95ca8a07a150ecf2 P f73de5713f8f45968cc13181138452b0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20251023 20:15:37.414587 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:37.414421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:58203)
1023 20:15:37.414514 (+ 93us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:37.414554 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20251023 20:15:38.382694 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:38.382387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:44569)
1023 20:15:38.382574 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:38.382637 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20251023 20:15:38.414438 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:38.414183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:53683)
1023 20:15:38.414330 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:38.414379 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20251023 20:15:38.415356 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:38.415157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:58425)
1023 20:15:38.415278 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:38.415311 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20251023 20:15:39.383541 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:39.383286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:49597)
1023 20:15:39.383441 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:39.383482 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20251023 20:15:39.415172 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:39.414929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:58333)
1023 20:15:39.415083 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:39.415116 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20251023 20:15:39.416021 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:39.415839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:49129)
1023 20:15:39.415948 (+ 109us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:39.415981 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
I20251023 20:15:40.384573 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:40.384263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:60629)
1023 20:15:40.384441 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:40.384491 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20251023 20:15:40.416131 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:40.415827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:41031)
1023 20:15:40.416017 (+ 190us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:40.416068 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20251023 20:15:40.416862 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:40.416662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:39605)
1023 20:15:40.416792 (+ 130us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:40.416833 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20251023 20:15:40.416903 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:40.416682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:37607)
1023 20:15:40.416825 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:40.416860 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20251023 20:15:41.385504 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:41.385195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:37217)
1023 20:15:41.385399 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:41.385447 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20251023 20:15:41.417580 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:41.417326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:37981)
1023 20:15:41.417503 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:41.417534 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20251023 20:15:41.417598 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:41.417323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:55109)
1023 20:15:41.417474 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:41.417516 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20251023 20:15:42.386438 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:42.386155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:46697)
1023 20:15:42.386328 (+ 173us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:42.386379 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20251023 20:15:42.418390 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:42.418137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:55045)
1023 20:15:42.418293 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:42.418335 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20251023 20:15:42.418390 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:42.418133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:46009)
1023 20:15:42.418299 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:42.418335 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20251023 20:15:43.387419 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:43.387123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:47055)
1023 20:15:43.387310 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:43.387362 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20251023 20:15:43.419289 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:43.419025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:44039)
1023 20:15:43.419210 (+ 185us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:43.419240 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20251023 20:15:43.419288 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:43.419025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:53661)
1023 20:15:43.419185 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:43.419233 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20251023 20:15:44.388386 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:44.388082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:60871)
1023 20:15:44.388276 (+ 194us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:44.388327 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20251023 20:15:44.419938 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:44.419687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:58539)
1023 20:15:44.419830 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:44.419876 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20251023 20:15:44.419999 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:44.419751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:57729)
1023 20:15:44.419920 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:44.419955 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
I20251023 20:15:45.389386 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:45.389125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:59789)
1023 20:15:45.389283 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:45.389321 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20251023 20:15:45.420917 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:45.420587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:38767)
1023 20:15:45.420813 (+ 226us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:45.420860 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20251023 20:15:45.420933 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:45.420587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:60799)
1023 20:15:45.420837 (+ 250us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:45.420883 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
I20251023 20:15:46.390369 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:46.390061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:50551)
1023 20:15:46.390258 (+ 197us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:46.390310 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20251023 20:15:46.421710 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:46.421443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:34413)
1023 20:15:46.421601 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:46.421644 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20251023 20:15:46.421710 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:46.421448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:42265)
1023 20:15:46.421629 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:46.421664 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20251023 20:15:47.391499 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:47.391148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:55759)
1023 20:15:47.391379 (+ 231us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:47.391437 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
I20251023 20:15:47.422752 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:47.422433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:43359)
1023 20:15:47.422616 (+ 183us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:47.422676 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20251023 20:15:47.422852 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:47.422429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:43611)
1023 20:15:47.422748 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:47.422788 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":227,"threads_started":1}
I20251023 20:15:48.392536 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:48.392227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:39757)
1023 20:15:48.392431 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:48.392479 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20251023 20:15:48.423687 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:48.423444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:57333)
1023 20:15:48.423587 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:48.423628 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20251023 20:15:48.423748 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:48.423471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:40775)
1023 20:15:48.423661 (+ 190us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:48.423697 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20251023 20:15:49.393517 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:49.393246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.193:33085)
1023 20:15:49.393421 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:49.393459 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20251023 20:15:49.424657 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:49.424338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.195:37401)
1023 20:15:49.424541 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:49.424579 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
I20251023 20:15:49.424660 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:49.424338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:43961 (local address 127.9.117.194:57065)
1023 20:15:49.424565 (+ 227us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:49.424613 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
I20251023 20:15:49.943677 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 25307
I20251023 20:15:49.949172 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 25470
I20251023 20:15:49.954883 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 25605
2025-10-23T20:15:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19675 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:15:50 dist-test-slave-x62p krb5kdc[25804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:15:50 dist-test-slave-x62p krb5kdc[25804](info): set up 2 sockets
Oct 23 20:15:50 dist-test-slave-x62p krb5kdc[25804](info): commencing operation
krb5kdc: starting...
W20251023 20:15:52.069438 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.091s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:15:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:52.236419 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37775
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:52.323657 25820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:52.323935 25820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:52.323979 25820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:52.325845 25820 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251023 20:15:52.325928 25820 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251023 20:15:52.325955 25820 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251023 20:15:52.325976 25820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:52.326002 25820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:52.326040 25820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:52.326071 25820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:52.326103 25820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:52.328195 25820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:52.328543 25820 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:52.328819 25820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:52.331661 25828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:52.331682 25825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:52.331748 25826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:52.332005 25820 server_base.cc:1047] running on GCE node
I20251023 20:15:52.332247 25820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:52.332530 25820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:52.333732 25820 hybrid_clock.cc:648] HybridClock initialized: now 1761250552333694 us; error 48 us; skew 500 ppm
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:52.336601 25820 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:15:52.337474 25820 webserver.cc:492] Webserver started at http://127.9.117.254:43301/ using document root <none> and password file <none>
I20251023 20:15:52.337776 25820 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:52.337980 25820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:52.338127 25820 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:52.339252 25820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "f74cc2fc7cff4eaeb21057ecbe55004b"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "eb5a208c89116f1e1f27764d604155db"
server_key_iv: "4de188b494f40383254136e0ba424b8d"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.339622 25820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "f74cc2fc7cff4eaeb21057ecbe55004b"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "eb5a208c89116f1e1f27764d604155db"
server_key_iv: "4de188b494f40383254136e0ba424b8d"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.341101 25820 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:15:52.341984 25835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.342191 25820 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:52.342267 25820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "f74cc2fc7cff4eaeb21057ecbe55004b"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "eb5a208c89116f1e1f27764d604155db"
server_key_iv: "4de188b494f40383254136e0ba424b8d"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.342345 25820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:52.355839 25820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:52.357934 25820 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:52.358059 25820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:52.362632 25820 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37775
I20251023 20:15:52.362671 25887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37775 every 8 connection(s)
I20251023 20:15:52.363051 25820 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:52.363786 25888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:52.366636 25888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Bootstrap starting.
I20251023 20:15:52.367300 25888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:52.367579 25888 log.cc:826] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:52.368400 25888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: No bootstrap required, opened a new log
I20251023 20:15:52.369967 25888 raft_consensus.cc:359] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:15:52.370131 25888 raft_consensus.cc:385] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:52.370157 25888 raft_consensus.cc:740] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cc2fc7cff4eaeb21057ecbe55004b, State: Initialized, Role: FOLLOWER
I20251023 20:15:52.370289 25888 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:15:52.370366 25888 raft_consensus.cc:399] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:15:52.370398 25888 raft_consensus.cc:493] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:15:52.370436 25888 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:52.371119 25888 raft_consensus.cc:515] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:15:52.371234 25888 leader_election.cc:304] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74cc2fc7cff4eaeb21057ecbe55004b; no voters:
I20251023 20:15:52.371423 25888 leader_election.cc:290] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:15:52.371510 25891 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:52.371670 25891 raft_consensus.cc:697] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 1 LEADER]: Becoming Leader. State: Replica: f74cc2fc7cff4eaeb21057ecbe55004b, State: Running, Role: LEADER
I20251023 20:15:52.371693 25888 sys_catalog.cc:565] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:52.371953 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 25820
I20251023 20:15:52.372063 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:15:52.372238 9687 external_mini_cluster.cc:1468] Setting key c1700aa6a33b4534350d5c674a6b7ff1
I20251023 20:15:52.372494 25891 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:15:52.373317 25892 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } } }
I20251023 20:15:52.373328 25893 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74cc2fc7cff4eaeb21057ecbe55004b. Latest consensus state: current_term: 1 leader_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } } }
I20251023 20:15:52.373703 25892 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:52.373713 25893 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: This master's current role is: LEADER
W20251023 20:15:52.374095 25907 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:15:52.374143 25907 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:15:52.374261 25910 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:52.374444 25910 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:52.376538 25910 catalog_manager.cc:1357] Generated new cluster ID: 1bc89d0d7fe54a1cb41fa32a86dd3f0c
I20251023 20:15:52.376713 25910 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:52.386662 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:52.372984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.9.117.254:37775)
1023 20:15:52.374393 (+ 1409us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:52.374407 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:52.374432 (+ 25us) server_negotiation.cc:408] Connection header received
1023 20:15:52.374571 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:52.374576 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:52.374631 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:52.374923 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:52.376079 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.377339 (+ 1260us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:52.378268 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.378548 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:52.381749 (+ 3201us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:52.381761 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:52.381762 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:52.381802 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:52.384294 (+ 2492us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.384980 (+ 686us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.384985 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.384986 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.385081 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.385416 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.385418 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.385419 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.385811 (+ 392us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:52.385994 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:52.386211 (+ 217us) server_negotiation.cc:300] Negotiation successful
1023 20:15:52.386408 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1357,"thread_start_us":108,"threads_started":1}
I20251023 20:15:52.391947 25910 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:15:52.392535 25910 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:15:52.399529 25910 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Generated new TSK 0
I20251023 20:15:52.399780 25910 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:52.464082 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37775
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:52.550385 25916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:52.550601 25916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:52.550637 25916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:52.552292 25916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:52.552362 25916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:52.552400 25916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:15:52.554417 25916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:52.554708 25916 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:52.554942 25916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:52.557890 25922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:52.557916 25924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:52.558004 25921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:52.558004 25916 server_base.cc:1047] running on GCE node
I20251023 20:15:52.558326 25916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:52.558547 25916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:52.559731 25916 hybrid_clock.cc:648] HybridClock initialized: now 1761250552559721 us; error 37 us; skew 500 ppm
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:52.562613 25916 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:15:52.563298 25916 webserver.cc:492] Webserver started at http://127.9.117.193:42641/ using document root <none> and password file <none>
I20251023 20:15:52.563505 25916 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:52.563547 25916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:52.563692 25916 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:52.564803 25916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "49f86bf31f6e435e94e1b0265114c561"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "66210326357b1483bef70c04c8feb56f"
server_key_iv: "446d6b8ea1a5c376396dcbab1aa1495d"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.565156 25916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "49f86bf31f6e435e94e1b0265114c561"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "66210326357b1483bef70c04c8feb56f"
server_key_iv: "446d6b8ea1a5c376396dcbab1aa1495d"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.566524 25916 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:15:52.567371 25931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.567638 25916 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:52.567725 25916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "49f86bf31f6e435e94e1b0265114c561"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "66210326357b1483bef70c04c8feb56f"
server_key_iv: "446d6b8ea1a5c376396dcbab1aa1495d"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.567803 25916 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:52.582319 25916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:52.584474 25916 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:52.584654 25916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:52.584934 25916 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:52.585300 25916 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:52.585335 25916 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.585374 25916 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:52.585403 25916 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.591950 25916 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:45485
I20251023 20:15:52.592062 26044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:45485 every 8 connection(s)
I20251023 20:15:52.592344 25916 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:52.599272 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 25916
I20251023 20:15:52.599390 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:15:52.599584 9687 external_mini_cluster.cc:1468] Setting key 4c0b290c1f513ea994dd262ee2d49f45
I20251023 20:15:52.601824 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:52.593139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:56961 (local address 127.9.117.254:37775)
1023 20:15:52.593192 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:52.593194 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:52.593350 (+ 156us) server_negotiation.cc:408] Connection header received
1023 20:15:52.593879 (+ 529us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:52.593880 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:52.593895 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:52.593964 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:52.594867 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.595383 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:52.595959 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.596084 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:52.598716 (+ 2632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:52.598722 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:52.598723 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:52.598732 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:52.600382 (+ 1650us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.600877 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.600878 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.600878 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.600907 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.601086 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.601087 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.601087 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.601227 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:52.601295 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:52.601543 (+ 248us) server_negotiation.cc:300] Negotiation successful
1023 20:15:52.601665 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:15:52.601889 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:52.593088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:56961)
1023 20:15:52.593232 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:52.593265 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:52.593800 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:52.594023 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:52.594026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:52.594375 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:52.594712 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:52.594716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.595465 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:52.595466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:52.595882 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:52.595883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.596000 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:52.596983 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:52.597003 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:52.598605 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:52.600475 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:52.600488 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:52.600492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:52.600796 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:52.600971 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:52.600972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:52.600972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:52.601037 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:52.601283 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:52.601286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:52.601418 (+ 132us) client_negotiation.cc:769] Sending connection context
1023 20:15:52.601512 (+ 94us) client_negotiation.cc:240] Negotiation successful
1023 20:15:52.601646 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20251023 20:15:52.602579 26045 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37775
I20251023 20:15:52.602684 26045 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:52.602901 26045 heartbeater.cc:507] Master 127.9.117.254:37775 requested a full tablet report, sending...
I20251023 20:15:52.603404 25852 ts_manager.cc:194] Registered new tserver with Master: 49f86bf31f6e435e94e1b0265114c561 (127.9.117.193:45485)
I20251023 20:15:52.604176 25852 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:56961
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:52.648221 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37775
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:52.735222 26052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:52.735460 26052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:52.735492 26052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:52.737185 26052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:52.737265 26052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:52.737323 26052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:15:52.739837 26052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:52.740142 26052 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:52.740376 26052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:52.743577 26058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:52.743585 26060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:52.743623 26057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:52.743851 26052 server_base.cc:1047] running on GCE node
I20251023 20:15:52.744033 26052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:52.744264 26052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:52.745448 26052 hybrid_clock.cc:648] HybridClock initialized: now 1761250552745433 us; error 32 us; skew 500 ppm
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:52.748355 26052 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:15:52.749060 26052 webserver.cc:492] Webserver started at http://127.9.117.194:45861/ using document root <none> and password file <none>
I20251023 20:15:52.749279 26052 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:52.749315 26052 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:52.749432 26052 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:52.750521 26052 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "0285ec2f90e64172a0d90a77d2725b4e"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "fbf248f27658f83cfa5ce8b16d0840de"
server_key_iv: "4a522d0f6e905e3c0cdae728e4388704"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.750859 26052 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "0285ec2f90e64172a0d90a77d2725b4e"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "fbf248f27658f83cfa5ce8b16d0840de"
server_key_iv: "4a522d0f6e905e3c0cdae728e4388704"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.752283 26052 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:15:52.753185 26067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.753386 26052 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:52.753471 26052 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "0285ec2f90e64172a0d90a77d2725b4e"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "fbf248f27658f83cfa5ce8b16d0840de"
server_key_iv: "4a522d0f6e905e3c0cdae728e4388704"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.753540 26052 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:52.766640 26052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:52.769196 26052 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:52.769333 26052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:52.769555 26052 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:52.769934 26052 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:52.769972 26052 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.769999 26052 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:52.770011 26052 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.776208 26052 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:46123
I20251023 20:15:52.776314 26180 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:46123 every 8 connection(s)
I20251023 20:15:52.776662 26052 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:52.783568 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 26052
I20251023 20:15:52.783679 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:15:52.783855 9687 external_mini_cluster.cc:1468] Setting key d1d862d85c72d216d076c29b47226af4
I20251023 20:15:52.786852 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:52.777564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:33395 (local address 127.9.117.254:37775)
1023 20:15:52.777659 (+ 95us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:52.777664 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:52.777969 (+ 305us) server_negotiation.cc:408] Connection header received
1023 20:15:52.778542 (+ 573us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:52.778543 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:52.778558 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:52.778619 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:52.779508 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.780020 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:52.780598 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.780810 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:52.783331 (+ 2521us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:52.783337 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:52.783338 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:52.783347 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:52.785320 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.785778 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.785780 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.785780 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.785812 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.786074 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.786075 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.786075 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.786225 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:52.786296 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:52.786645 (+ 349us) server_negotiation.cc:300] Negotiation successful
1023 20:15:52.786756 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251023 20:15:52.787025 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:52.777494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:33395)
1023 20:15:52.777807 (+ 313us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:52.777839 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:52.778434 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:52.778699 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:52.778704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:52.779083 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:52.779397 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:52.779400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.780111 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:52.780112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:52.780512 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:52.780514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.780672 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:52.781719 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:52.781740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:52.783230 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:52.785431 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:52.785440 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:52.785443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:52.785691 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:52.785916 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:52.785917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:52.785918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:52.786007 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:52.786331 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:52.786340 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:52.786498 (+ 158us) client_negotiation.cc:769] Sending connection context
1023 20:15:52.786607 (+ 109us) client_negotiation.cc:240] Negotiation successful
1023 20:15:52.786794 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":137,"threads_started":1}
I20251023 20:15:52.787572 26181 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37775
I20251023 20:15:52.787695 26181 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:52.787945 26181 heartbeater.cc:507] Master 127.9.117.254:37775 requested a full tablet report, sending...
I20251023 20:15:52.788385 25852 ts_manager.cc:194] Registered new tserver with Master: 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
I20251023 20:15:52.789037 25852 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:33395
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:15:52.835417 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37775
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:15:52.922168 26188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:52.922367 26188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:52.922394 26188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:52.924000 26188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:15:52.924063 26188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:52.924095 26188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:15:52.925884 26188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:15:52.926124 26188 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:52.926297 26188 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:52.929330 26193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:52.929330 26196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:52.929503 26188 server_base.cc:1047] running on GCE node
W20251023 20:15:52.929379 26194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:52.929777 26188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:52.930033 26188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:52.931214 26188 hybrid_clock.cc:648] HybridClock initialized: now 1761250552931209 us; error 55 us; skew 500 ppm
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:15:52.934463 26188 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:15:52.935184 26188 webserver.cc:492] Webserver started at http://127.9.117.195:36125/ using document root <none> and password file <none>
I20251023 20:15:52.935380 26188 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:52.935415 26188 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:52.935504 26188 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:52.936514 26188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "4eec434ee63c4c4e8563fb9e7ca707f2"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "dcef81db7d0f9c8fc11c2a4d5d51111e"
server_key_iv: "6705a2f51fb8812342bfe99fa602a6d9"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.936888 26188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "4eec434ee63c4c4e8563fb9e7ca707f2"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "dcef81db7d0f9c8fc11c2a4d5d51111e"
server_key_iv: "6705a2f51fb8812342bfe99fa602a6d9"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.938292 26188 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:52.939083 26203 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.939282 26188 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:15:52.939363 26188 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "4eec434ee63c4c4e8563fb9e7ca707f2"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "dcef81db7d0f9c8fc11c2a4d5d51111e"
server_key_iv: "6705a2f51fb8812342bfe99fa602a6d9"
server_key_version: "encryptionkey@0"
I20251023 20:15:52.939435 26188 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:52.955394 26188 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:52.958304 26188 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:52.958461 26188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:52.958726 26188 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:52.959076 26188 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:52.959112 26188 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.959151 26188 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:52.959182 26188 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:52.965584 26188 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:34073
I20251023 20:15:52.965656 26316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:34073 every 8 connection(s)
I20251023 20:15:52.965989 26188 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:15:52.970777 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 26188
I20251023 20:15:52.970881 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:15:52.971045 9687 external_mini_cluster.cc:1468] Setting key f6c5abf15725b6a5eb360067777b3b34
Oct 23 20:15:52 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250552, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:15:52.976801 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:52.966811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:36355 (local address 127.9.117.254:37775)
1023 20:15:52.966881 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:52.966884 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:52.967056 (+ 172us) server_negotiation.cc:408] Connection header received
1023 20:15:52.967599 (+ 543us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:52.967601 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:52.967615 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:52.967685 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:52.968532 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.969108 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:52.969675 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.969850 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:52.972660 (+ 2810us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:52.972663 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:52.972663 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:52.972670 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:52.974397 (+ 1727us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.975269 (+ 872us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.975272 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.975272 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.975312 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.975692 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.975694 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.975694 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.975830 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:52.975885 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:52.976544 (+ 659us) server_negotiation.cc:300] Negotiation successful
1023 20:15:52.976668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251023 20:15:52.976989 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:52.966751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:36355)
1023 20:15:52.966944 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:52.966972 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:52.967500 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:52.967763 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:52.967767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:52.968125 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:15:52.968427 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:52.968430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.969202 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:52.969203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:52.969592 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:52.969594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.969734 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:52.970740 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:15:52.970758 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:15:52.972538 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:15:52.974756 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:52.974771 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:52.974776 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:52.975114 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:52.975461 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:15:52.975463 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:15:52.975463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:15:52.975582 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:15:52.975937 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:15:52.975942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:15:52.976347 (+ 405us) client_negotiation.cc:769] Sending connection context
1023 20:15:52.976497 (+ 150us) client_negotiation.cc:240] Negotiation successful
1023 20:15:52.976693 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20251023 20:15:52.977995 26317 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37775
I20251023 20:15:52.978144 26317 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:52.978585 26317 heartbeater.cc:507] Master 127.9.117.254:37775 requested a full tablet report, sending...
I20251023 20:15:52.979296 25852 ts_manager.cc:194] Registered new tserver with Master: 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073)
I20251023 20:15:52.979893 25852 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:36355
I20251023 20:15:52.982959 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:15:52.993664 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:52.984763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.9.117.254:37775)
1023 20:15:52.984834 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:52.984836 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:52.984866 (+ 30us) server_negotiation.cc:408] Connection header received
1023 20:15:52.984920 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:52.984921 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:52.984933 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:52.985007 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:15:52.986179 (+ 1172us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.986984 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:52.987942 (+ 958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:52.988162 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:52.989378 (+ 1216us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:15:52.989385 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:15:52.989385 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:15:52.989415 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:15:52.991642 (+ 2227us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.992627 (+ 985us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.992630 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.992631 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.992666 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:15:52.992918 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:15:52.992919 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:15:52.992919 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:15:52.993119 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:15:52.993238 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:52.993416 (+ 178us) server_negotiation.cc:300] Negotiation successful
1023 20:15:52.993548 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:15:52.995850 25852 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:52.997947 25852 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:53.004654 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.000580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34073 (local address 127.0.0.1:52756)
1023 20:15:53.000710 (+ 130us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:53.000771 (+ 61us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:53.000861 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:53.001590 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:53.001591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:53.001593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:53.001832 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.001835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.003211 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.003212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:53.004235 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.004238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.004318 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.004386 (+ 68us) client_negotiation.cc:769] Sending connection context
1023 20:15:53.004452 (+ 66us) client_negotiation.cc:240] Negotiation successful
1023 20:15:53.004503 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20251023 20:15:53.004861 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.000586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:46123 (local address 127.0.0.1:57062)
1023 20:15:53.001025 (+ 439us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:53.001036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:53.001110 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:53.001431 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:53.001434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:53.001437 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:53.001743 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.001747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.003533 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.003534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:53.004428 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.004431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.004479 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.004484 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:15:53.004508 (+ 24us) client_negotiation.cc:240] Negotiation successful
1023 20:15:53.004564 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":92,"threads_started":1}
I20251023 20:15:53.005283 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.000798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.9.117.195:34073)
1023 20:15:53.001247 (+ 449us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:53.001262 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:53.001273 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:15:53.001453 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:53.001457 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:53.001478 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:53.001691 (+ 213us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:53.001936 (+ 245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.003118 (+ 1182us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.004363 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.005040 (+ 677us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.005112 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:53.005146 (+ 34us) server_negotiation.cc:300] Negotiation successful
1023 20:15:53.005196 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":105,"threads_started":1}
I20251023 20:15:53.005999 26251 tablet_service.cc:1505] Processing CreateTablet for tablet 25a22cb2893a419291a07a80bb4fc7af (DEFAULT_TABLE table=test-table [id=9681369584b548fab30722d75c304fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:53.006178 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.000856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.9.117.194:46123)
1023 20:15:53.001041 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:53.001050 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:53.001117 (+ 67us) server_negotiation.cc:408] Connection header received
1023 20:15:53.001195 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:53.001199 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:53.001219 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:53.001292 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:53.001846 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.003422 (+ 1576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.005120 (+ 1698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.005754 (+ 634us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.005957 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:53.005997 (+ 40us) server_negotiation.cc:300] Negotiation successful
1023 20:15:53.006070 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20251023 20:15:53.006378 26251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a22cb2893a419291a07a80bb4fc7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:53.006681 26115 tablet_service.cc:1505] Processing CreateTablet for tablet 25a22cb2893a419291a07a80bb4fc7af (DEFAULT_TABLE table=test-table [id=9681369584b548fab30722d75c304fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:53.006983 26115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a22cb2893a419291a07a80bb4fc7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:53.010054 26334 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Bootstrap starting.
I20251023 20:15:53.010787 26334 tablet_bootstrap.cc:654] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:53.010890 26335 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Bootstrap starting.
I20251023 20:15:53.011144 26334 log.cc:826] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:53.011706 26335 tablet_bootstrap.cc:654] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:53.011924 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.000822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:45485 (local address 127.0.0.1:54962)
1023 20:15:53.001249 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:53.001259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:53.001322 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:53.001603 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:53.001604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:53.001607 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:53.002078 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.002080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.005315 (+ 3235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.005316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:53.011615 (+ 6299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.011623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.011728 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.011734 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:15:53.011768 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:15:53.011826 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":87,"threads_started":1}
I20251023 20:15:53.012015 26334 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: No bootstrap required, opened a new log
I20251023 20:15:53.012070 26335 log.cc:826] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:53.012090 26334 ts_tablet_manager.cc:1403] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:53.012939 26335 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: No bootstrap required, opened a new log
I20251023 20:15:53.013018 26335 ts_tablet_manager.cc:1403] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:53.013090 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.001059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.9.117.193:45485)
1023 20:15:53.001235 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:53.001243 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:53.001381 (+ 138us) server_negotiation.cc:408] Connection header received
1023 20:15:53.001427 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:53.001429 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:53.001445 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:53.001505 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:53.002175 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.004644 (+ 2469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.011853 (+ 7209us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.012829 (+ 976us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.012899 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:53.012935 (+ 36us) server_negotiation.cc:300] Negotiation successful
1023 20:15:53.012993 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20251023 20:15:53.013759 25979 tablet_service.cc:1505] Processing CreateTablet for tablet 25a22cb2893a419291a07a80bb4fc7af (DEFAULT_TABLE table=test-table [id=9681369584b548fab30722d75c304fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:53.013974 26334 raft_consensus.cc:359] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.014094 25979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a22cb2893a419291a07a80bb4fc7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:53.014140 26334 raft_consensus.cc:385] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:53.014180 26334 raft_consensus.cc:740] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec434ee63c4c4e8563fb9e7ca707f2, State: Initialized, Role: FOLLOWER
I20251023 20:15:53.014297 26334 consensus_queue.cc:260] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.014652 26317 heartbeater.cc:499] Master 127.9.117.254:37775 was elected leader, sending a full tablet report...
I20251023 20:15:53.014791 26334 ts_tablet_manager.cc:1434] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:15:53.016222 26335 raft_consensus.cc:359] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.016433 26335 raft_consensus.cc:385] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:53.016476 26335 raft_consensus.cc:740] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0285ec2f90e64172a0d90a77d2725b4e, State: Initialized, Role: FOLLOWER
I20251023 20:15:53.016763 26335 consensus_queue.cc:260] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.017217 26181 heartbeater.cc:499] Master 127.9.117.254:37775 was elected leader, sending a full tablet report...
I20251023 20:15:53.017778 26340 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Bootstrap starting.
I20251023 20:15:53.018174 26335 ts_tablet_manager.cc:1434] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251023 20:15:53.018527 26340 tablet_bootstrap.cc:654] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:53.018893 26340 log.cc:826] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:53.019810 26340 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: No bootstrap required, opened a new log
I20251023 20:15:53.019891 26340 ts_tablet_manager.cc:1403] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:53.021454 26340 raft_consensus.cc:359] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.021615 26340 raft_consensus.cc:385] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:53.021651 26340 raft_consensus.cc:740] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f86bf31f6e435e94e1b0265114c561, State: Initialized, Role: FOLLOWER
I20251023 20:15:53.021775 26340 consensus_queue.cc:260] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.022135 26045 heartbeater.cc:499] Master 127.9.117.254:37775 was elected leader, sending a full tablet report...
I20251023 20:15:53.022240 26340 ts_tablet_manager.cc:1434] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20251023 20:15:53.027318 26182 tablet.cc:2378] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:15:53.093096 26046 tablet.cc:2378] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:53.211103 26342 raft_consensus.cc:493] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:53.211241 26342 raft_consensus.cc:515] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.211575 26342 leader_election.cc:290] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073), 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
I20251023 20:15:53.214083 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.211795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:46123 (local address 127.9.117.193:46373)
1023 20:15:53.211957 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:53.211968 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:53.212042 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:53.212199 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:53.212200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:53.212202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:53.212462 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.212465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.213198 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.213199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:53.213800 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.213802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.213880 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.213884 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:15:53.213917 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:15:53.213956 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20251023 20:15:53.214087 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.211796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34073 (local address 127.9.117.193:46015)
1023 20:15:53.211989 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:53.211999 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:53.212105 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:53.212271 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:15:53.212272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:15:53.212273 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:15:53.212491 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.212493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.213179 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.213181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:15:53.213773 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:15:53.213775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.213836 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.213840 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:15:53.213884 (+ 44us) client_negotiation.cc:240] Negotiation successful
1023 20:15:53.213924 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":14,"thread_start_us":75,"threads_started":1}
I20251023 20:15:53.214510 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.211862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:46015 (local address 127.9.117.195:34073)
1023 20:15:53.211926 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:53.211928 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:53.212099 (+ 171us) server_negotiation.cc:408] Connection header received
1023 20:15:53.212182 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:53.212184 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:53.212196 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:53.212260 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:53.212569 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.213100 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.213861 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.214363 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.214373 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:53.214393 (+ 20us) server_negotiation.cc:300] Negotiation successful
1023 20:15:53.214422 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251023 20:15:53.214632 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.211917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:46373 (local address 127.9.117.194:46123)
1023 20:15:53.211974 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:53.211976 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:53.212047 (+ 71us) server_negotiation.cc:408] Connection header received
1023 20:15:53.212116 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:53.212118 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:53.212128 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:53.212185 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:15:53.212538 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.213111 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.213910 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.214387 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.214396 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:53.214421 (+ 25us) server_negotiation.cc:300] Negotiation successful
1023 20:15:53.214545 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:15:53.214834 26271 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25a22cb2893a419291a07a80bb4fc7af" candidate_uuid: "49f86bf31f6e435e94e1b0265114c561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" is_pre_election: true
I20251023 20:15:53.214918 26135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25a22cb2893a419291a07a80bb4fc7af" candidate_uuid: "49f86bf31f6e435e94e1b0265114c561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0285ec2f90e64172a0d90a77d2725b4e" is_pre_election: true
I20251023 20:15:53.214993 26271 raft_consensus.cc:2468] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f86bf31f6e435e94e1b0265114c561 in term 0.
I20251023 20:15:53.215077 26135 raft_consensus.cc:2468] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f86bf31f6e435e94e1b0265114c561 in term 0.
I20251023 20:15:53.215219 25932 leader_election.cc:304] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f86bf31f6e435e94e1b0265114c561, 4eec434ee63c4c4e8563fb9e7ca707f2; no voters:
I20251023 20:15:53.215374 26342 raft_consensus.cc:2804] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:53.215442 26342 raft_consensus.cc:493] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:53.215461 26342 raft_consensus.cc:3060] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:53.216141 26342 raft_consensus.cc:515] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.216277 26342 leader_election.cc:290] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 1 election: Requested vote from peers 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073), 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
I20251023 20:15:53.216462 26135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25a22cb2893a419291a07a80bb4fc7af" candidate_uuid: "49f86bf31f6e435e94e1b0265114c561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0285ec2f90e64172a0d90a77d2725b4e"
I20251023 20:15:53.216486 26271 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25a22cb2893a419291a07a80bb4fc7af" candidate_uuid: "49f86bf31f6e435e94e1b0265114c561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2"
I20251023 20:15:53.216547 26271 raft_consensus.cc:3060] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:53.216549 26135 raft_consensus.cc:3060] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:53.216559 26318 tablet.cc:2378] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:53.217294 26271 raft_consensus.cc:2468] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f86bf31f6e435e94e1b0265114c561 in term 1.
I20251023 20:15:53.217370 26135 raft_consensus.cc:2468] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f86bf31f6e435e94e1b0265114c561 in term 1.
I20251023 20:15:53.217485 25932 leader_election.cc:304] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f86bf31f6e435e94e1b0265114c561, 4eec434ee63c4c4e8563fb9e7ca707f2; no voters:
I20251023 20:15:53.217574 26342 raft_consensus.cc:2804] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:53.217741 26342 raft_consensus.cc:697] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 1 LEADER]: Becoming Leader. State: Replica: 49f86bf31f6e435e94e1b0265114c561, State: Running, Role: LEADER
I20251023 20:15:53.217857 26342 consensus_queue.cc:237] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:15:53.218608 25852 catalog_manager.cc:5649] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49f86bf31f6e435e94e1b0265114c561 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "49f86bf31f6e435e94e1b0265114c561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:53.232936 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.230768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.9.117.193:45485)
1023 20:15:53.230853 (+ 85us) server_negotiation.cc:207] Beginning negotiation
1023 20:15:53.230855 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:15:53.230864 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:15:53.230917 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:15:53.230918 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:15:53.230935 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:15:53.230990 (+ 55us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:15:53.231257 (+ 267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.231744 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:15:53.232323 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:15:53.232483 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:15:53.232526 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:15:53.232639 (+ 113us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:15:53.232752 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1023 20:15:53.232819 (+ 67us) server_negotiation.cc:300] Negotiation successful
1023 20:15:53.232847 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251023 20:15:53.234717 26271 raft_consensus.cc:1275] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 1 FOLLOWER]: Refusing update from remote peer 49f86bf31f6e435e94e1b0265114c561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:53.234717 26135 raft_consensus.cc:1275] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 1 FOLLOWER]: Refusing update from remote peer 49f86bf31f6e435e94e1b0265114c561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:53.235044 26345 consensus_queue.cc:1048] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:53.235178 26342 consensus_queue.cc:1048] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:53.237681 26348 mvcc.cc:204] Tried to move back new op lower bound from 7214082266046599168 to 7214082265980911616. Current Snapshot: MvccSnapshot[applied={T|T < 7214082266046599168}]
I20251023 20:15:53.238706 26349 mvcc.cc:204] Tried to move back new op lower bound from 7214082266046599168 to 7214082265980911616. Current Snapshot: MvccSnapshot[applied={T|T < 7214082266046599168}]
I20251023 20:15:53.240058 25851 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36878:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:15:53.240173 25851 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36878:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:15:53.240979 25851 catalog_manager.cc:5936] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Sending DeleteTablet for 3 replicas of tablet 25a22cb2893a419291a07a80bb4fc7af
I20251023 20:15:53.241353 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 25916
I20251023 20:15:53.241410 26251 tablet_service.cc:1552] Processing DeleteTablet for tablet 25a22cb2893a419291a07a80bb4fc7af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:53 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:52756
I20251023 20:15:53.241448 26115 tablet_service.cc:1552] Processing DeleteTablet for tablet 25a22cb2893a419291a07a80bb4fc7af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:53 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:57062
I20251023 20:15:53.242213 26358 tablet_replica.cc:333] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: stopping tablet replica
I20251023 20:15:53.242331 26358 raft_consensus.cc:2243] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:53.242457 26358 raft_consensus.cc:2272] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:53.242934 26358 ts_tablet_manager.cc:1916] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:53.245220 26358 ts_tablet_manager.cc:1929] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:53.245296 26358 log.cc:1199] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/25a22cb2893a419291a07a80bb4fc7af
I20251023 20:15:53.245589 26358 ts_tablet_manager.cc:1950] T 25a22cb2893a419291a07a80bb4fc7af P 0285ec2f90e64172a0d90a77d2725b4e: Deleting consensus metadata
I20251023 20:15:53.246091 25838 catalog_manager.cc:4985] TS 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123): tablet 25a22cb2893a419291a07a80bb4fc7af (table test-table [id=9681369584b548fab30722d75c304fd9]) successfully deleted
I20251023 20:15:53.248956 26357 tablet_replica.cc:333] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: stopping tablet replica
W20251023 20:15:53.249012 25836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:45485 (error 108)
I20251023 20:15:53.249094 26357 raft_consensus.cc:2243] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:53.249125 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 26052
I20251023 20:15:53.249239 26357 raft_consensus.cc:2272] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:53.250638 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.250470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:45485 (local address 127.0.0.1:54976)
1023 20:15:53.250534 (+ 64us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:53.250584 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:45485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251023 20:15:53.250774 25836 catalog_manager.cc:4712] TS 49f86bf31f6e435e94e1b0265114c561 (127.9.117.193:45485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25a22cb2893a419291a07a80bb4fc7af: Network error: Client connection negotiation failed: client connection to 127.9.117.193:45485: connect: Connection refused (error 111)
I20251023 20:15:53.253803 26357 ts_tablet_manager.cc:1916] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:53.254498 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 26188
I20251023 20:15:53.259783 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 25820
I20251023 20:15:53.259930 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:15:53.259759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34073 (local address 127.0.0.1:52764)
1023 20:15:53.259833 (+ 74us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:53.259871 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.195:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20251023 20:16:13.266106 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37775
--webserver_interface=127.9.117.254
--webserver_port=43301
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:13.351788 26360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:13.352003 26360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:13.352032 26360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:13.353767 26360 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251023 20:16:13.353837 26360 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251023 20:16:13.353852 26360 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251023 20:16:13.353863 26360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:16:13.353878 26360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:13.353891 26360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:13.353904 26360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:16:13.353916 26360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:16:13.356316 26360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=43301
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:13.356640 26360 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:13.356911 26360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:16:13.364289 26360 server_base.cc:1047] running on GCE node
W20251023 20:16:13.364317 26369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:13.364317 26366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:13.364635 26367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:13.365098 26360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:13.365417 26360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:13.366596 26360 hybrid_clock.cc:648] HybridClock initialized: now 1761250573366557 us; error 55 us; skew 500 ppm
Oct 23 20:16:13 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:13.369700 26360 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:16:13.370424 26360 webserver.cc:492] Webserver started at http://127.9.117.254:43301/ using document root <none> and password file <none>
I20251023 20:16:13.370656 26360 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:13.370718 26360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:13.372421 26360 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:13.373517 26376 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:13.373839 26360 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:16:13.373946 26360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "f74cc2fc7cff4eaeb21057ecbe55004b"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "eb5a208c89116f1e1f27764d604155db"
server_key_iv: "4de188b494f40383254136e0ba424b8d"
server_key_version: "encryptionkey@0"
I20251023 20:16:13.374359 26360 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:13.392176 26360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:13.394775 26360 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:13.394953 26360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:13.400399 26360 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37775
I20251023 20:16:13.400444 26428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37775 every 8 connection(s)
I20251023 20:16:13.400905 26360 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:16:13.402001 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 26360
I20251023 20:16:13.402314 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:45485
--local_ip_for_outbound_sockets=127.9.117.193
--tserver_master_addrs=127.9.117.254:37775
--webserver_port=42641
--webserver_interface=127.9.117.193
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:16:13.402335 26429 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:16:13.403170 26429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Bootstrap starting.
I20251023 20:16:13.414597 26429 log.cc:826] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:13.417120 26429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:16:13.417464 26429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Bootstrap complete.
I20251023 20:16:13.419909 26429 raft_consensus.cc:359] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:16:13.420159 26429 raft_consensus.cc:740] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cc2fc7cff4eaeb21057ecbe55004b, State: Initialized, Role: FOLLOWER
I20251023 20:16:13.420395 26429 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:16:13.420516 26429 raft_consensus.cc:399] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:16:13.420557 26429 raft_consensus.cc:493] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:16:13.420948 26429 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:16:13.422691 26429 raft_consensus.cc:515] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:16:13.422825 26429 leader_election.cc:304] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74cc2fc7cff4eaeb21057ecbe55004b; no voters:
I20251023 20:16:13.423116 26429 leader_election.cc:290] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [CANDIDATE]: Term 2 election: Requested vote from peers
I20251023 20:16:13.423177 26433 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:16:13.423367 26433 raft_consensus.cc:697] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 2 LEADER]: Becoming Leader. State: Replica: f74cc2fc7cff4eaeb21057ecbe55004b, State: Running, Role: LEADER
I20251023 20:16:13.423429 26429 sys_catalog.cc:565] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:16:13.423518 26433 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:16:13.423966 26435 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74cc2fc7cff4eaeb21057ecbe55004b. Latest consensus state: current_term: 2 leader_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } } }
I20251023 20:16:13.423992 26434 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } } }
I20251023 20:16:13.424077 26435 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:13.424084 26434 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:13.424336 26438 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:16:13.425400 26438 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:16:13.425570 26438 catalog_manager.cc:1269] Loaded cluster ID: 1bc89d0d7fe54a1cb41fa32a86dd3f0c
I20251023 20:16:13.425851 26438 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:16:13.427244 26438 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:16:13.427446 26438 catalog_manager.cc:6033] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Loaded TSK: 0
I20251023 20:16:13.440821 26438 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Generated new TSK 1
I20251023 20:16:13.440979 26438 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251023 20:16:13.496016 26431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:13.496235 26431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:13.496264 26431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:13.497932 26431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:13.497992 26431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:13.498023 26431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:16:13.499815 26431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:45485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=42641
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:13.500061 26431 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:13.500276 26431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:13.503253 26459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:13.503319 26456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:13.503469 26457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:13.503695 26431 server_base.cc:1047] running on GCE node
I20251023 20:16:13.503890 26431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:13.504137 26431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:13.505312 26431 hybrid_clock.cc:648] HybridClock initialized: now 1761250573505295 us; error 30 us; skew 500 ppm
Oct 23 20:16:13 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:13.508891 26431 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:16:13.509692 26431 webserver.cc:492] Webserver started at http://127.9.117.193:42641/ using document root <none> and password file <none>
I20251023 20:16:13.509946 26431 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:13.510016 26431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:13.511826 26431 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:16:13.512717 26466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:13.513074 26431 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:13.513159 26431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "49f86bf31f6e435e94e1b0265114c561"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "66210326357b1483bef70c04c8feb56f"
server_key_iv: "446d6b8ea1a5c376396dcbab1aa1495d"
server_key_version: "encryptionkey@0"
I20251023 20:16:13.513475 26431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:13.528532 26431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:13.530800 26431 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:13.530927 26431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:13.531194 26431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:13.531682 26473 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251023 20:16:13.532996 26431 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251023 20:16:13.533070 26431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20251023 20:16:13.533118 26431 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251023 20:16:13.533790 26431 ts_tablet_manager.cc:616] Registered 1 tablets
I20251023 20:16:13.533831 26431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251023 20:16:13.533928 26473 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Bootstrap starting.
I20251023 20:16:13.542923 26431 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:45485
I20251023 20:16:13.543339 26431 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:16:13.548508 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 26431
I20251023 20:16:13.548964 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:46123
--local_ip_for_outbound_sockets=127.9.117.194
--tserver_master_addrs=127.9.117.254:37775
--webserver_port=45861
--webserver_interface=127.9.117.194
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:16:13.549501 26473 log.cc:826] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:13.553340 26580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:45485 every 8 connection(s)
Oct 23 20:16:13 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:13.556649 26473 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:16:13.557200 26473 tablet_bootstrap.cc:492] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Bootstrap complete.
I20251023 20:16:13.558691 26473 ts_tablet_manager.cc:1403] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.009s
I20251023 20:16:13.560212 26473 raft_consensus.cc:359] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.560575 26473 raft_consensus.cc:740] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f86bf31f6e435e94e1b0265114c561, State: Initialized, Role: FOLLOWER
I20251023 20:16:13.560770 26473 consensus_queue.cc:260] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.561108 26473 ts_tablet_manager.cc:1434] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:16:13.561722 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.544384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:55077 (local address 127.9.117.254:37775)
1023 20:16:13.549873 (+ 5489us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.549887 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.549925 (+ 38us) server_negotiation.cc:408] Connection header received
1023 20:16:13.550067 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.550075 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.550133 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.550310 (+ 177us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:13.551591 (+ 1281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.552837 (+ 1246us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.553715 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.553990 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.557211 (+ 3221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:13.557238 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:13.557239 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:13.557295 (+ 56us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:13.559672 (+ 2377us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.560195 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.560198 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.560199 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.560271 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.560478 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.560479 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.560479 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.560787 (+ 308us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:13.560894 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.561346 (+ 452us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.561491 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5421,"thread_start_us":106,"threads_started":1}
I20251023 20:16:13.561731 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.544319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:55077)
1023 20:16:13.544997 (+ 678us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:13.545039 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:13.545899 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:13.550519 (+ 4620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:13.550524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:13.551012 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:13.551430 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.551436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.552973 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.552976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:13.553607 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.553610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.553768 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.554475 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:13.554511 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:13.556596 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:13.559809 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:13.559819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:13.559823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:13.560095 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:13.560347 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:13.560348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:13.560348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:13.560418 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:13.560952 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:13.560958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:13.561135 (+ 177us) client_negotiation.cc:769] Sending connection context
1023 20:16:13.561271 (+ 136us) client_negotiation.cc:240] Negotiation successful
1023 20:16:13.561482 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":110,"threads_started":1}
I20251023 20:16:13.562525 26581 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37775
I20251023 20:16:13.562642 26581 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:13.562863 26581 heartbeater.cc:507] Master 127.9.117.254:37775 requested a full tablet report, sending...
I20251023 20:16:13.563591 26393 ts_manager.cc:194] Registered new tserver with Master: 49f86bf31f6e435e94e1b0265114c561 (127.9.117.193:45485)
I20251023 20:16:13.564683 26393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:55077
I20251023 20:16:13.565558 26581 heartbeater.cc:499] Master 127.9.117.254:37775 was elected leader, sending a full tablet report...
I20251023 20:16:13.568665 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.564219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:45485 (local address 127.0.0.1:58674)
1023 20:16:13.565126 (+ 907us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:13.565800 (+ 674us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:13.565887 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:13.566192 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:13.566195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:13.566197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:13.566424 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.566426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.567356 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.567358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:13.568304 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.568308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.568406 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.568453 (+ 47us) client_negotiation.cc:769] Sending connection context
1023 20:16:13.568505 (+ 52us) client_negotiation.cc:240] Negotiation successful
1023 20:16:13.568552 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":115,"threads_started":1}
I20251023 20:16:13.569464 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.564273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.9.117.193:45485)
1023 20:16:13.565092 (+ 819us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.565102 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.565957 (+ 855us) server_negotiation.cc:408] Connection header received
1023 20:16:13.565997 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.566000 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.566017 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.566082 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:13.566523 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.567268 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.568865 (+ 1597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.569304 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.569350 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.569378 (+ 28us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.569409 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":88,"threads_started":1}
I20251023 20:16:13.569808 26497 tablet_service.cc:1552] Processing DeleteTablet for tablet 25a22cb2893a419291a07a80bb4fc7af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:53 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:58674
I20251023 20:16:13.570382 26590 tablet_replica.cc:333] stopping tablet replica
I20251023 20:16:13.570482 26590 raft_consensus.cc:2243] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:13.570555 26590 raft_consensus.cc:2272] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:13.570880 26590 ts_tablet_manager.cc:1916] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:13.572970 26590 ts_tablet_manager.cc:1929] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:13.573063 26590 log.cc:1199] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal/wals/25a22cb2893a419291a07a80bb4fc7af
I20251023 20:16:13.573341 26590 ts_tablet_manager.cc:1950] T 25a22cb2893a419291a07a80bb4fc7af P 49f86bf31f6e435e94e1b0265114c561: Deleting consensus metadata
I20251023 20:16:13.573931 26377 catalog_manager.cc:4985] TS 49f86bf31f6e435e94e1b0265114c561 (127.9.117.193:45485): tablet 25a22cb2893a419291a07a80bb4fc7af (table test-table [id=9681369584b548fab30722d75c304fd9]) successfully deleted
W20251023 20:16:13.666203 26586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:13.666421 26586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:13.666453 26586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:13.668103 26586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:13.668164 26586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:13.668195 26586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:16:13.670102 26586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:46123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=45861
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:13.670357 26586 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:13.670539 26586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:13.673580 26595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:13.673669 26596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:13.673704 26586 server_base.cc:1047] running on GCE node
W20251023 20:16:13.673669 26598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:13.674084 26586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:13.674321 26586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:13.675503 26586 hybrid_clock.cc:648] HybridClock initialized: now 1761250573675477 us; error 49 us; skew 500 ppm
Oct 23 20:16:13 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:13.678367 26586 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:16:13.678974 26586 webserver.cc:492] Webserver started at http://127.9.117.194:45861/ using document root <none> and password file <none>
I20251023 20:16:13.679185 26586 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:13.679239 26586 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:13.680791 26586 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:16:13.681470 26605 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:13.681661 26586 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:13.681742 26586 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "0285ec2f90e64172a0d90a77d2725b4e"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "fbf248f27658f83cfa5ce8b16d0840de"
server_key_iv: "4a522d0f6e905e3c0cdae728e4388704"
server_key_version: "encryptionkey@0"
I20251023 20:16:13.682072 26586 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:13.700404 26586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:13.702718 26586 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:13.702880 26586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:13.703137 26586 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:13.703470 26586 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:13.703506 26586 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:13.703544 26586 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:13.703577 26586 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:13.710150 26586 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:46123
I20251023 20:16:13.710216 26718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:46123 every 8 connection(s)
I20251023 20:16:13.710552 26586 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:16:13.715752 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 26586
I20251023 20:16:13.716076 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:34073
--local_ip_for_outbound_sockets=127.9.117.195
--tserver_master_addrs=127.9.117.254:37775
--webserver_port=36125
--webserver_interface=127.9.117.195
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 23 20:16:13 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:13.720865 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.711311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:47643 (local address 127.9.117.254:37775)
1023 20:16:13.711386 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.711389 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.711558 (+ 169us) server_negotiation.cc:408] Connection header received
1023 20:16:13.712136 (+ 578us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.712138 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.712163 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.712248 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:13.713157 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.713680 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.714257 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.714431 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.717308 (+ 2877us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:13.717314 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:13.717315 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:13.717327 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:13.719269 (+ 1942us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.719764 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.719766 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.719767 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.719805 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.720023 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.720023 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.720024 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.720186 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:13.720266 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.720530 (+ 264us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.720676 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251023 20:16:13.720916 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.711278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:47643)
1023 20:16:13.711447 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:13.711476 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:13.712028 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:13.712326 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:13.712330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:13.712675 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:13.713053 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.713057 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.713774 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.713775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:13.714178 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.714180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.714332 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.715346 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:13.715365 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:13.717166 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:13.719382 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:13.719393 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:13.719396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:13.719663 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:13.719880 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:13.719881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:13.719882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:13.719963 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:13.720245 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:13.720248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:13.720389 (+ 141us) client_negotiation.cc:769] Sending connection context
1023 20:16:13.720483 (+ 94us) client_negotiation.cc:240] Negotiation successful
1023 20:16:13.720652 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20251023 20:16:13.721493 26719 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37775
I20251023 20:16:13.721735 26719 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:13.722077 26719 heartbeater.cc:507] Master 127.9.117.254:37775 requested a full tablet report, sending...
I20251023 20:16:13.722594 26393 ts_manager.cc:194] Registered new tserver with Master: 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
I20251023 20:16:13.723021 26393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:47643
W20251023 20:16:13.809336 26722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:13.809576 26722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:13.809607 26722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:13.811447 26722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:13.811532 26722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:13.811568 26722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:16:13.813699 26722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:34073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=36125
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:13.813987 26722 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:13.814159 26722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:13.817310 26730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:13.817322 26727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:13.817473 26722 server_base.cc:1047] running on GCE node
W20251023 20:16:13.817343 26728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:13.817888 26722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:13.818127 26722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:13.819299 26722 hybrid_clock.cc:648] HybridClock initialized: now 1761250573819279 us; error 30 us; skew 500 ppm
Oct 23 20:16:13 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:13.822609 26722 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:16:13.823427 26722 webserver.cc:492] Webserver started at http://127.9.117.195:36125/ using document root <none> and password file <none>
I20251023 20:16:13.823657 26722 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:13.823719 26722 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:13.825390 26722 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:16:13.826305 26737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:13.826663 26722 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:13.826771 26722 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "4eec434ee63c4c4e8563fb9e7ca707f2"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "dcef81db7d0f9c8fc11c2a4d5d51111e"
server_key_iv: "6705a2f51fb8812342bfe99fa602a6d9"
server_key_version: "encryptionkey@0"
I20251023 20:16:13.827113 26722 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:13.848531 26722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:13.850699 26722 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:13.850859 26722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:13.851110 26722 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:13.851570 26744 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20251023 20:16:13.852511 26744 ts_tablet_manager.cc:1863] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20251023 20:16:13.852876 26744 ts_tablet_manager.cc:1916] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:13.854833 26744 ts_tablet_manager.cc:1929] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:13.854928 26744 log.cc:1199] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/25a22cb2893a419291a07a80bb4fc7af
I20251023 20:16:13.855517 26744 ts_tablet_manager.cc:1950] T 25a22cb2893a419291a07a80bb4fc7af P 4eec434ee63c4c4e8563fb9e7ca707f2: Deleting consensus metadata
I20251023 20:16:13.855768 26722 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20251023 20:16:13.855844 26722 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20251023 20:16:13.855890 26722 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:13.855923 26722 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:13.870898 26722 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:34073
I20251023 20:16:13.871191 26851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:34073 every 8 connection(s)
I20251023 20:16:13.871292 26722 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:16:13.872958 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 26722
Oct 23 20:16:13 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:13.888159 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.873423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.9.117.254:37775)
1023 20:16:13.873739 (+ 316us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.873741 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.873764 (+ 23us) server_negotiation.cc:408] Connection header received
1023 20:16:13.873784 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.873785 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.873795 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.873850 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:13.874722 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.875262 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.876650 (+ 1388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.877394 (+ 744us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.884302 (+ 6908us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:13.884309 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:13.884309 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:13.884320 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:13.886686 (+ 2366us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.887198 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.887200 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.887200 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.887232 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.887570 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.887571 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.887571 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.887703 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:13.887878 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.887971 (+ 93us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.888074 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":62,"threads_started":1}
I20251023 20:16:13.888778 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.872224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:48883 (local address 127.9.117.254:37775)
1023 20:16:13.872297 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.872300 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.872543 (+ 243us) server_negotiation.cc:408] Connection header received
1023 20:16:13.873022 (+ 479us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.873023 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.873038 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.873094 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:13.874196 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.874772 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.875557 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.875764 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.884573 (+ 8809us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:13.884583 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:13.884584 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:13.884591 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:13.887082 (+ 2491us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.887687 (+ 605us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.887688 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.887689 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.887715 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.887998 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.887999 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.888000 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.888114 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:13.888279 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.888565 (+ 286us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.888661 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:16:13.888991 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.872183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:48883)
1023 20:16:13.872384 (+ 201us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:13.872417 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:13.872919 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:13.873211 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:13.873216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:13.873667 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:13.874067 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.874072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.874908 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.874909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:13.875370 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.875371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.875527 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.876524 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:13.876629 (+ 105us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:13.884476 (+ 7847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:13.887208 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:13.887227 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:13.887232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:13.887551 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:13.887800 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:13.887801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:13.887802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:13.887926 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:13.888187 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:13.888194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:13.888409 (+ 215us) client_negotiation.cc:769] Sending connection context
1023 20:16:13.888546 (+ 137us) client_negotiation.cc:240] Negotiation successful
1023 20:16:13.888760 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
I20251023 20:16:13.889442 26852 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37775
I20251023 20:16:13.889588 26852 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:13.889825 26852 heartbeater.cc:507] Master 127.9.117.254:37775 requested a full tablet report, sending...
I20251023 20:16:13.890308 26393 ts_manager.cc:194] Registered new tserver with Master: 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073)
I20251023 20:16:13.890714 26393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:48883
I20251023 20:16:13.901672 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:16:13.907236 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.902895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.9.117.254:37775)
1023 20:16:13.903058 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.903059 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.903068 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:16:13.903234 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.903235 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.903274 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.903342 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:16:13.903946 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.905861 (+ 1915us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.906769 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.906909 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.906962 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:16:13.907023 (+ 61us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
1023 20:16:13.907170 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33240: token expired
Metrics: {"server-negotiator.queue_time_us":136}
W20251023 20:16:13.907303 26584 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33240: token expired
W20251023 20:16:13.907979 26323 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.117.254:37775)
I20251023 20:16:13.914518 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.908260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.9.117.254:37775)
1023 20:16:13.908322 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.908324 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.908335 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:16:13.908410 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.908411 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.908423 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.908475 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:13.909208 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.909678 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.910246 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.910437 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.911617 (+ 1180us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:13.911622 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:13.911623 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:13.911631 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:13.913389 (+ 1758us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.913760 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.913761 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.913762 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.913788 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:13.913933 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:13.913934 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:13.913934 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:13.914074 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:13.914184 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.914288 (+ 104us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.914397 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251023 20:16:13.915783 26393 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:16:13.917541 26393 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:16:13.920393 26497 tablet_service.cc:1505] Processing CreateTablet for tablet d7d9b57df86e4200a956ef71a07a385f (DEFAULT_TABLE table=test-table [id=23246b15933e45e5839b82c83b9a037a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:13.920650 26497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d9b57df86e4200a956ef71a07a385f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:13.922789 26473 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Bootstrap starting.
I20251023 20:16:13.923215 26473 tablet_bootstrap.cc:654] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:13.923449 26858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.920047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:46123 (local address 127.0.0.1:53548)
1023 20:16:13.920299 (+ 252us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:13.920308 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:13.920363 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:13.920636 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:13.920637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:13.920640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:13.921082 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.921087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.922185 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.922186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:13.923157 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.923160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.923231 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.923238 (+ 7us) client_negotiation.cc:769] Sending connection context
1023 20:16:13.923303 (+ 65us) client_negotiation.cc:240] Negotiation successful
1023 20:16:13.923349 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
I20251023 20:16:13.924222 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.920283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.9.117.194:46123)
1023 20:16:13.920447 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.920458 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.920468 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:13.920507 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.920510 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.920524 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.920611 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:13.921177 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.922101 (+ 924us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.923257 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.923931 (+ 674us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.924007 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.924054 (+ 47us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.924112 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20251023 20:16:13.924325 26473 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: No bootstrap required, opened a new log
I20251023 20:16:13.924367 26473 ts_tablet_manager.cc:1403] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:16:13.924477 26473 raft_consensus.cc:359] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.924556 26473 raft_consensus.cc:385] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:13.924592 26473 raft_consensus.cc:740] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f86bf31f6e435e94e1b0265114c561, State: Initialized, Role: FOLLOWER
I20251023 20:16:13.924638 26473 consensus_queue.cc:260] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.924767 26473 ts_tablet_manager.cc:1434] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:13.924925 26653 tablet_service.cc:1505] Processing CreateTablet for tablet d7d9b57df86e4200a956ef71a07a385f (DEFAULT_TABLE table=test-table [id=23246b15933e45e5839b82c83b9a037a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:13.925251 26653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d9b57df86e4200a956ef71a07a385f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:13.925742 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.919936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34073 (local address 127.0.0.1:51700)
1023 20:16:13.920064 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:13.920076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:13.920113 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:13.923082 (+ 2969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:13.923083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:13.923087 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:13.923274 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.923276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.924573 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.924574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:13.925488 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.925490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.925587 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.925592 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:16:13.925627 (+ 35us) client_negotiation.cc:240] Negotiation successful
1023 20:16:13.925688 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20251023 20:16:13.926401 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.920088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.9.117.195:34073)
1023 20:16:13.922854 (+ 2766us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.922863 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.922879 (+ 16us) server_negotiation.cc:408] Connection header received
1023 20:16:13.922932 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.922935 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.922953 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.923060 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:13.923395 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.924494 (+ 1099us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.925592 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.926178 (+ 586us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.926243 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.926277 (+ 34us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.926326 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2709,"thread_start_us":109,"threads_started":1}
I20251023 20:16:13.926934 26786 tablet_service.cc:1505] Processing CreateTablet for tablet d7d9b57df86e4200a956ef71a07a385f (DEFAULT_TABLE table=test-table [id=23246b15933e45e5839b82c83b9a037a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:13.927268 26786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d9b57df86e4200a956ef71a07a385f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:13.928927 26862 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Bootstrap starting.
I20251023 20:16:13.930027 26862 tablet_bootstrap.cc:654] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:13.930368 26862 log.cc:826] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:13.930555 26744 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: Bootstrap starting.
I20251023 20:16:13.931217 26744 tablet_bootstrap.cc:654] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:13.931483 26862 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: No bootstrap required, opened a new log
I20251023 20:16:13.931556 26744 log.cc:826] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:13.931586 26862 ts_tablet_manager.cc:1403] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:16:13.932446 26744 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: No bootstrap required, opened a new log
I20251023 20:16:13.932523 26744 ts_tablet_manager.cc:1403] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:16:13.933674 26862 raft_consensus.cc:359] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.933869 26862 raft_consensus.cc:385] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:13.933907 26862 raft_consensus.cc:740] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0285ec2f90e64172a0d90a77d2725b4e, State: Initialized, Role: FOLLOWER
I20251023 20:16:13.934024 26862 consensus_queue.cc:260] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.934350 26862 ts_tablet_manager.cc:1434] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:16:13.934357 26719 heartbeater.cc:499] Master 127.9.117.254:37775 was elected leader, sending a full tablet report...
I20251023 20:16:13.934458 26744 raft_consensus.cc:359] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.934598 26744 raft_consensus.cc:385] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:13.934623 26744 raft_consensus.cc:740] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec434ee63c4c4e8563fb9e7ca707f2, State: Initialized, Role: FOLLOWER
I20251023 20:16:13.934710 26744 consensus_queue.cc:260] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.935037 26744 ts_tablet_manager.cc:1434] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:16:13.935076 26852 heartbeater.cc:499] Master 127.9.117.254:37775 was elected leader, sending a full tablet report...
W20251023 20:16:13.961154 26720 tablet.cc:2378] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:16:13.986066 26587 raft_consensus.cc:493] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:13.986222 26587 raft_consensus.cc:515] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.986593 26587 leader_election.cc:290] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073), 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
I20251023 20:16:13.989163 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.986810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:46123 (local address 127.9.117.193:56969)
1023 20:16:13.986878 (+ 68us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:13.986894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:13.986954 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:13.987159 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:13.987160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:13.987162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:13.987341 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.987343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.988088 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.988090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:13.988897 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.988901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.988985 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.988990 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:16:13.989025 (+ 35us) client_negotiation.cc:240] Negotiation successful
1023 20:16:13.989066 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20251023 20:16:13.989166 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.986811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34073 (local address 127.9.117.193:41097)
1023 20:16:13.987033 (+ 222us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:13.987044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:13.987137 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:13.987282 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:13.987282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:13.987284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:13.987466 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.987468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.988180 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.988182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:13.988852 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:13.988855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.988942 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.988948 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:16:13.988983 (+ 35us) client_negotiation.cc:240] Negotiation successful
1023 20:16:13.989030 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":19,"thread_start_us":117,"threads_started":1}
I20251023 20:16:13.989594 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.986889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:41097 (local address 127.9.117.195:34073)
1023 20:16:13.986945 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.986948 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.987138 (+ 190us) server_negotiation.cc:408] Connection header received
1023 20:16:13.987206 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.987207 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.987219 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.987269 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:13.987544 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.988079 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.988946 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.989443 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.989453 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.989484 (+ 31us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.989522 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:16:13.989625 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:13.986969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:56969 (local address 127.9.117.194:46123)
1023 20:16:13.987024 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:13.987026 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:13.987036 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:13.987071 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:13.987072 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:13.987085 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:13.987148 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:13.987427 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.987974 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:13.988992 (+ 1018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:13.989479 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:13.989487 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:13.989513 (+ 26us) server_negotiation.cc:300] Negotiation successful
1023 20:16:13.989544 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251023 20:16:13.990001 26806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7d9b57df86e4200a956ef71a07a385f" candidate_uuid: "49f86bf31f6e435e94e1b0265114c561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" is_pre_election: true
I20251023 20:16:13.990078 26673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7d9b57df86e4200a956ef71a07a385f" candidate_uuid: "49f86bf31f6e435e94e1b0265114c561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0285ec2f90e64172a0d90a77d2725b4e" is_pre_election: true
I20251023 20:16:13.990163 26806 raft_consensus.cc:2468] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f86bf31f6e435e94e1b0265114c561 in term 0.
I20251023 20:16:13.990255 26673 raft_consensus.cc:2468] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f86bf31f6e435e94e1b0265114c561 in term 0.
I20251023 20:16:13.990437 26467 leader_election.cc:304] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f86bf31f6e435e94e1b0265114c561, 4eec434ee63c4c4e8563fb9e7ca707f2; no voters:
I20251023 20:16:13.990586 26587 raft_consensus.cc:2804] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:16:13.990667 26587 raft_consensus.cc:493] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:16:13.990701 26587 raft_consensus.cc:3060] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:13.991416 26587 raft_consensus.cc:515] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.991565 26587 leader_election.cc:290] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 1 election: Requested vote from peers 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073), 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
I20251023 20:16:13.991848 26806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7d9b57df86e4200a956ef71a07a385f" candidate_uuid: "49f86bf31f6e435e94e1b0265114c561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2"
I20251023 20:16:13.991951 26806 raft_consensus.cc:3060] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:13.991936 26673 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7d9b57df86e4200a956ef71a07a385f" candidate_uuid: "49f86bf31f6e435e94e1b0265114c561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0285ec2f90e64172a0d90a77d2725b4e"
I20251023 20:16:13.992038 26673 raft_consensus.cc:3060] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:13.992884 26673 raft_consensus.cc:2468] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f86bf31f6e435e94e1b0265114c561 in term 1.
I20251023 20:16:13.992882 26806 raft_consensus.cc:2468] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f86bf31f6e435e94e1b0265114c561 in term 1.
I20251023 20:16:13.993518 26469 leader_election.cc:304] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0285ec2f90e64172a0d90a77d2725b4e, 49f86bf31f6e435e94e1b0265114c561; no voters:
I20251023 20:16:13.993686 26587 raft_consensus.cc:2804] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:13.993904 26587 raft_consensus.cc:697] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 LEADER]: Becoming Leader. State: Replica: 49f86bf31f6e435e94e1b0265114c561, State: Running, Role: LEADER
I20251023 20:16:13.993991 26587 consensus_queue.cc:237] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:13.994910 26392 catalog_manager.cc:5649] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49f86bf31f6e435e94e1b0265114c561 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "49f86bf31f6e435e94e1b0265114c561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } health_report { overall_health: HEALTHY } } }
I20251023 20:16:14.018770 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:14.016438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.9.117.193:45485)
1023 20:16:14.016553 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:14.016556 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:14.016566 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:14.016615 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:14.016616 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:14.016636 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:14.016714 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:16:14.017059 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:14.017566 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:14.018197 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:14.018347 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:14.018391 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:16:14.018504 (+ 113us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:16:14.018585 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:14.018646 (+ 61us) server_negotiation.cc:300] Negotiation successful
1023 20:16:14.018675 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251023 20:16:14.020573 26673 raft_consensus.cc:1275] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 1 FOLLOWER]: Refusing update from remote peer 49f86bf31f6e435e94e1b0265114c561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:14.020573 26806 raft_consensus.cc:1275] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 1 FOLLOWER]: Refusing update from remote peer 49f86bf31f6e435e94e1b0265114c561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:14.021019 26587 consensus_queue.cc:1048] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:14.021190 26868 consensus_queue.cc:1048] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:14.024899 26872 mvcc.cc:204] Tried to move back new op lower bound from 7214082351185108992 to 7214082351080091648. Current Snapshot: MvccSnapshot[applied={T|T < 7214082351185108992 or (T in {7214082351185108992})}]
I20251023 20:16:14.025275 26871 mvcc.cc:204] Tried to move back new op lower bound from 7214082351185108992 to 7214082351080091648. Current Snapshot: MvccSnapshot[applied={T|T < 7214082351185108992}]
I20251023 20:16:14.026912 26393 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:14.027040 26393 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33246:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:14.027956 26393 catalog_manager.cc:5936] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Sending DeleteTablet for 3 replicas of tablet d7d9b57df86e4200a956ef71a07a385f
I20251023 20:16:14.028314 26786 tablet_service.cc:1552] Processing DeleteTablet for tablet d7d9b57df86e4200a956ef71a07a385f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:14 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:51700
I20251023 20:16:14.028365 26497 tablet_service.cc:1552] Processing DeleteTablet for tablet d7d9b57df86e4200a956ef71a07a385f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:14 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:58674
I20251023 20:16:14.028448 26653 tablet_service.cc:1552] Processing DeleteTablet for tablet d7d9b57df86e4200a956ef71a07a385f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:14 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:53548
I20251023 20:16:14.028651 26880 tablet_replica.cc:333] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: stopping tablet replica
I20251023 20:16:14.028681 26590 tablet_replica.cc:333] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: stopping tablet replica
I20251023 20:16:14.028780 26590 raft_consensus.cc:2243] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:16:14.028807 26881 tablet_replica.cc:333] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: stopping tablet replica
I20251023 20:16:14.028900 26590 raft_consensus.cc:2272] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:14.028956 26881 raft_consensus.cc:2243] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:14.028966 26880 raft_consensus.cc:2243] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:14.029107 26880 raft_consensus.cc:2272] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:14.029215 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 26431
I20251023 20:16:14.029104 26881 raft_consensus.cc:2272] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:14.031656 26880 ts_tablet_manager.cc:1916] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:14.031685 26881 ts_tablet_manager.cc:1916] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:14.033241 26881 ts_tablet_manager.cc:1929] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:14.033409 26881 log.cc:1199] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/d7d9b57df86e4200a956ef71a07a385f
I20251023 20:16:14.033785 26881 ts_tablet_manager.cc:1950] T d7d9b57df86e4200a956ef71a07a385f P 0285ec2f90e64172a0d90a77d2725b4e: Deleting consensus metadata
I20251023 20:16:14.033870 26880 ts_tablet_manager.cc:1929] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:14.033946 26880 log.cc:1199] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/d7d9b57df86e4200a956ef71a07a385f
I20251023 20:16:14.034185 26880 ts_tablet_manager.cc:1950] T d7d9b57df86e4200a956ef71a07a385f P 4eec434ee63c4c4e8563fb9e7ca707f2: Deleting consensus metadata
I20251023 20:16:14.034366 26379 catalog_manager.cc:4985] TS 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123): tablet d7d9b57df86e4200a956ef71a07a385f (table test-table [id=23246b15933e45e5839b82c83b9a037a]) successfully deleted
I20251023 20:16:14.034601 26377 catalog_manager.cc:4985] TS 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073): tablet d7d9b57df86e4200a956ef71a07a385f (table test-table [id=23246b15933e45e5839b82c83b9a037a]) successfully deleted
W20251023 20:16:14.035480 26377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:45485 (error 108)
I20251023 20:16:14.035868 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 26586
I20251023 20:16:14.036465 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:14.035921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:45485 (local address 127.0.0.1:58700)
1023 20:16:14.036389 (+ 468us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:14.036427 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:45485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445}
W20251023 20:16:14.036590 26377 catalog_manager.cc:4712] TS 49f86bf31f6e435e94e1b0265114c561 (127.9.117.193:45485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7d9b57df86e4200a956ef71a07a385f: Network error: Client connection negotiation failed: client connection to 127.9.117.193:45485: connect: Connection refused (error 111)
I20251023 20:16:14.041958 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 26722
I20251023 20:16:14.047041 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 26360
I20251023 20:16:34.053287 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37775
--webserver_interface=127.9.117.254
--webserver_port=43301
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:34.138986 26883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:34.139211 26883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:34.139250 26883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:34.141194 26883 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251023 20:16:34.141258 26883 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251023 20:16:34.141271 26883 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251023 20:16:34.141283 26883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:16:34.141299 26883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:34.141314 26883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:34.141342 26883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:16:34.141368 26883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:16:34.143270 26883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=43301
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:34.143561 26883 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:34.143761 26883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:34.147571 26888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:34.147610 26891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:34.147742 26883 server_base.cc:1047] running on GCE node
W20251023 20:16:34.147569 26889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:34.148115 26883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:34.148454 26883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:34.149636 26883 hybrid_clock.cc:648] HybridClock initialized: now 1761250594149615 us; error 38 us; skew 500 ppm
Oct 23 20:16:34 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250594, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:34.153342 26883 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:16:34.154161 26883 webserver.cc:492] Webserver started at http://127.9.117.254:43301/ using document root <none> and password file <none>
I20251023 20:16:34.154426 26883 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:34.154486 26883 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:34.155884 26883 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:34.156474 26898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.156647 26883 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.156718 26883 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "f74cc2fc7cff4eaeb21057ecbe55004b"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "eb5a208c89116f1e1f27764d604155db"
server_key_iv: "4de188b494f40383254136e0ba424b8d"
server_key_version: "encryptionkey@0"
I20251023 20:16:34.157058 26883 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:34.168972 26883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:34.171106 26883 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:34.171231 26883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:34.175508 26883 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37775
I20251023 20:16:34.175612 26950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37775 every 8 connection(s)
I20251023 20:16:34.175935 26883 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:16:34.177340 26951 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:16:34.177959 26951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Bootstrap starting.
I20251023 20:16:34.178599 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 26883
I20251023 20:16:34.178912 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:45485
--local_ip_for_outbound_sockets=127.9.117.193
--tserver_master_addrs=127.9.117.254:37775
--webserver_port=42641
--webserver_interface=127.9.117.193
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:16:34.189033 26951 log.cc:826] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:34.191313 26951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:16:34.191632 26951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Bootstrap complete.
I20251023 20:16:34.193889 26951 raft_consensus.cc:359] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:16:34.194087 26951 raft_consensus.cc:740] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cc2fc7cff4eaeb21057ecbe55004b, State: Initialized, Role: FOLLOWER
I20251023 20:16:34.194209 26951 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:16:34.194268 26951 raft_consensus.cc:399] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:16:34.194310 26951 raft_consensus.cc:493] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:16:34.194357 26951 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 2 FOLLOWER]: Advancing to term 3
I20251023 20:16:34.195286 26951 raft_consensus.cc:515] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:16:34.195382 26951 leader_election.cc:304] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74cc2fc7cff4eaeb21057ecbe55004b; no voters:
I20251023 20:16:34.195569 26951 leader_election.cc:290] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [CANDIDATE]: Term 3 election: Requested vote from peers
I20251023 20:16:34.195626 26955 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 3 FOLLOWER]: Leader election won for term 3
I20251023 20:16:34.195742 26955 raft_consensus.cc:697] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [term 3 LEADER]: Becoming Leader. State: Replica: f74cc2fc7cff4eaeb21057ecbe55004b, State: Running, Role: LEADER
I20251023 20:16:34.195851 26951 sys_catalog.cc:565] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:16:34.195833 26955 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } }
I20251023 20:16:34.196262 26956 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } } }
I20251023 20:16:34.196285 26957 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74cc2fc7cff4eaeb21057ecbe55004b. Latest consensus state: current_term: 3 leader_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74cc2fc7cff4eaeb21057ecbe55004b" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37775 } } }
I20251023 20:16:34.196448 26956 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:34.196471 26957 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:34.197129 26964 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:16:34.197958 26964 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:16:34.198065 26964 catalog_manager.cc:1269] Loaded cluster ID: 1bc89d0d7fe54a1cb41fa32a86dd3f0c
I20251023 20:16:34.198091 26964 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:16:34.199466 26964 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:16:34.199680 26964 catalog_manager.cc:6033] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Loaded TSK: 0
I20251023 20:16:34.199729 26964 catalog_manager.cc:6033] T 00000000000000000000000000000000 P f74cc2fc7cff4eaeb21057ecbe55004b: Loaded TSK: 1
F20251023 20:16:34.199939 26964 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f18f8e6adcd google::LogMessage::Fail() at ??:0
@ 0x7f18f8e6eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f18f8e6a7cc google::LogMessage::Flush() at ??:0
@ 0x7f18f8e6bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f18f76a95b6 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f18f76a6dc3 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f18fb6f467e kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f18fb6f489a kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f18fb6f49e3 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f18fb6b4707 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f18fb6bf503 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f18f9f6c2b5 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f18f9f624e7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f18fa2916db start_thread at ??:0
@ 0x7f18f83bf71f clone at ??:0
W20251023 20:16:34.294134 26953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:34.294414 26953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:34.294458 26953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:34.297055 26953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:34.297143 26953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:34.297197 26953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:16:34.300021 26953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:45485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=42641
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:34.300338 26953 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:34.300587 26953 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:34.303654 26995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:34.303880 26953 server_base.cc:1047] running on GCE node
W20251023 20:16:34.303647 26993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:34.303653 26992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:34.304216 26953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:34.304471 26953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:34.305652 26953 hybrid_clock.cc:648] HybridClock initialized: now 1761250594305636 us; error 40 us; skew 500 ppm
Oct 23 20:16:34 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250594, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:34.308859 26953 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:16:34.309453 26953 webserver.cc:492] Webserver started at http://127.9.117.193:42641/ using document root <none> and password file <none>
I20251023 20:16:34.309661 26953 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:34.309705 26953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:34.311038 26953 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:34.311758 27002 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.311950 26953 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:34.312031 26953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "49f86bf31f6e435e94e1b0265114c561"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "66210326357b1483bef70c04c8feb56f"
server_key_iv: "446d6b8ea1a5c376396dcbab1aa1495d"
server_key_version: "encryptionkey@0"
I20251023 20:16:34.312328 26953 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:34.348361 26953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:34.350893 26953 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:34.351054 26953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:34.351317 26953 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:34.351807 27009 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251023 20:16:34.353153 26953 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251023 20:16:34.353229 26953 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20251023 20:16:34.353274 26953 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251023 20:16:34.353942 26953 ts_tablet_manager.cc:616] Registered 1 tablets
I20251023 20:16:34.353981 26953 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251023 20:16:34.354198 27009 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Bootstrap starting.
I20251023 20:16:34.362378 26953 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:45485
I20251023 20:16:34.362571 27116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:45485 every 8 connection(s)
I20251023 20:16:34.362838 26953 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:16:34.365607 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 26953
I20251023 20:16:34.366112 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:46123
--local_ip_for_outbound_sockets=127.9.117.194
--tserver_master_addrs=127.9.117.254:37775
--webserver_port=45861
--webserver_interface=127.9.117.194
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:16:34.367841 27009 log.cc:826] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:34.370496 27009 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:16:34.370877 27009 tablet_bootstrap.cc:492] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Bootstrap complete.
I20251023 20:16:34.371896 27009 ts_tablet_manager.cc:1403] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20251023 20:16:34.372985 27009 raft_consensus.cc:359] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:34.373229 27009 raft_consensus.cc:740] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f86bf31f6e435e94e1b0265114c561, State: Initialized, Role: FOLLOWER
I20251023 20:16:34.373355 27009 consensus_queue.cc:260] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:34.373696 27009 ts_tablet_manager.cc:1434] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20251023 20:16:34.474063 27119 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:16:34.363677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:35871)
1023 20:16:34.363848 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.363884 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:34.364405 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:34.473876 (+109471us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
W20251023 20:16:34.474316 27117 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:37775 (0 consecutive failures): Network error: Failed to ping master at 127.9.117.254:37775: Client connection negotiation failed: client connection to 127.9.117.254:37775: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:16:34.474704 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:34.474576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:36355)
1023 20:16:34.474648 (+ 72us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.474683 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20251023 20:16:34.475019 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:34.474945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:43631)
1023 20:16:34.474984 (+ 39us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.475004 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20251023 20:16:34.475121 27117 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251023 20:16:34.480237 27120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:34.480422 27120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:34.480463 27120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:34.482036 27120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:34.482081 27120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:34.482129 27120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:16:34.483749 27120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:46123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=45861
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:34.484011 27120 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:34.484203 27120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:34.486869 27128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:34.486898 27130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:34.486874 27127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:34.487040 27120 server_base.cc:1047] running on GCE node
I20251023 20:16:34.487274 27120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:34.487457 27120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:34.488603 27120 hybrid_clock.cc:648] HybridClock initialized: now 1761250594488589 us; error 30 us; skew 500 ppm
Oct 23 20:16:34 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250594, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:34.491207 27120 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:16:34.491734 27120 webserver.cc:492] Webserver started at http://127.9.117.194:45861/ using document root <none> and password file <none>
I20251023 20:16:34.491902 27120 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:34.491935 27120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:34.493274 27120 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:34.493862 27137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.494082 27120 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:34.494163 27120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "0285ec2f90e64172a0d90a77d2725b4e"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "fbf248f27658f83cfa5ce8b16d0840de"
server_key_iv: "4a522d0f6e905e3c0cdae728e4388704"
server_key_version: "encryptionkey@0"
I20251023 20:16:34.494444 27120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:34.512696 27120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:34.514973 27120 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:34.515095 27120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:34.515328 27120 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:34.515679 27120 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:34.515717 27120 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.515741 27120 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:34.515758 27120 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.521867 27120 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:46123
I20251023 20:16:34.521960 27250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:46123 every 8 connection(s)
I20251023 20:16:34.522253 27120 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:16:34.522461 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 27120
I20251023 20:16:34.522814 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:34073
--local_ip_for_outbound_sockets=127.9.117.195
--tserver_master_addrs=127.9.117.254:37775
--webserver_port=36125
--webserver_interface=127.9.117.195
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:16:34.523407 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:34.523050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:59677)
1023 20:16:34.523202 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.523280 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
W20251023 20:16:34.523622 27251 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:37775 (0 consecutive failures): Network error: Failed to ping master at 127.9.117.254:37775: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
I20251023 20:16:34.523901 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:34.523804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:55821)
1023 20:16:34.523853 (+ 49us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.523878 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20251023 20:16:34.524173 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:34.524097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:52613)
1023 20:16:34.524133 (+ 36us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.524156 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20251023 20:16:34.524256 27251 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251023 20:16:34.605484 27254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:34.605705 27254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:34.605729 27254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:34.607334 27254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:34.607383 27254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:34.607415 27254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:16:34.609196 27254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:34073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=36125
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:34.609443 27254 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:34.609653 27254 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:34.612519 27259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:34.612596 27254 server_base.cc:1047] running on GCE node
W20251023 20:16:34.612586 27262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:34.612582 27260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:34.613042 27254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:34.613268 27254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:34.614418 27254 hybrid_clock.cc:648] HybridClock initialized: now 1761250594614404 us; error 30 us; skew 500 ppm
Oct 23 20:16:34 dist-test-slave-x62p krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250594, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:34.617416 27254 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:16:34.618024 27254 webserver.cc:492] Webserver started at http://127.9.117.195:36125/ using document root <none> and password file <none>
I20251023 20:16:34.618237 27254 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:34.618280 27254 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:34.619736 27254 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:34.620395 27269 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.620568 27254 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:16:34.620649 27254 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "4eec434ee63c4c4e8563fb9e7ca707f2"
format_stamp: "Formatted at 2025-10-23 20:15:52 on dist-test-slave-x62p"
server_key: "dcef81db7d0f9c8fc11c2a4d5d51111e"
server_key_iv: "6705a2f51fb8812342bfe99fa602a6d9"
server_key_version: "encryptionkey@0"
I20251023 20:16:34.620997 27254 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:34.636368 27254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:34.638495 27254 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:34.638652 27254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:34.638903 27254 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:34.639218 27254 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:34.639253 27254 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.639289 27254 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:34.639318 27254 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:34.645610 27254 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:34073
I20251023 20:16:34.645712 27382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:34073 every 8 connection(s)
I20251023 20:16:34.645998 27254 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:16:34.647002 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:34.646636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:51449)
1023 20:16:34.646784 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.646865 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20251023 20:16:34.647219 27383 heartbeater.cc:646] Failed to heartbeat to 127.9.117.254:37775 (0 consecutive failures): Network error: Failed to ping master at 127.9.117.254:37775: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
I20251023 20:16:34.647498 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:34.647392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:40391)
1023 20:16:34.647443 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.647466 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20251023 20:16:34.647776 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:34.647674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:58401)
1023 20:16:34.647711 (+ 37us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:34.647734 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20251023 20:16:34.647943 27383 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251023 20:16:34.647957 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 27254
I20251023 20:16:35.475860 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:35.475569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:55753)
1023 20:16:35.475758 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:35.475804 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
I20251023 20:16:35.524966 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:35.524666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:55113)
1023 20:16:35.524862 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:35.524906 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":130,"threads_started":1}
I20251023 20:16:35.648533 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:35.648298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:55803)
1023 20:16:35.648435 (+ 137us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:35.648479 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20251023 20:16:35.892176 27390 raft_consensus.cc:493] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:35.892339 27390 raft_consensus.cc:515] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:35.892709 27390 leader_election.cc:290] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073), 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
Oct 23 20:16:35 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250594, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.195@KRBTEST.COM
Oct 23 20:16:35 dist-test-slave-x62p krb5kdc[25804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250594, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.194@KRBTEST.COM
I20251023 20:16:35.903841 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:35.893092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:49539 (local address 127.9.117.194:46123)
1023 20:16:35.893253 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:35.893273 (+ 20us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:35.893298 (+ 25us) server_negotiation.cc:408] Connection header received
1023 20:16:35.893839 (+ 541us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:35.893842 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:35.893879 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:35.893989 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:35.895027 (+ 1038us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:35.896237 (+ 1210us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:35.896928 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:35.897198 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:35.900172 (+ 2974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:35.900181 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:35.900181 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:35.900206 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:35.902179 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:35.902709 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:35.902713 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:35.902714 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:35.902773 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:35.902980 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:35.902981 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:35.902981 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:35.903219 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:35.903311 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:35.903590 (+ 279us) server_negotiation.cc:300] Negotiation successful
1023 20:16:35.903717 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20251023 20:16:35.903833 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:35.892939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34073 (local address 127.9.117.193:60947)
1023 20:16:35.893081 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:35.893093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:35.893170 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:35.894032 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:35.894036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:35.894558 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:35.894893 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:35.894898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:35.896352 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:35.896353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:35.896821 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:35.896825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:35.896975 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:35.897863 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:35.897867 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:35.899719 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:35.902214 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:35.902225 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:35.902228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:35.902614 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:35.902837 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:35.902838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:35.902838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:35.902920 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:35.903288 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:35.903294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:35.903421 (+ 127us) client_negotiation.cc:769] Sending connection context
1023 20:16:35.903525 (+ 104us) client_negotiation.cc:240] Negotiation successful
1023 20:16:35.903721 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":17,"thread_start_us":63,"threads_started":1}
I20251023 20:16:35.903841 27393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:35.893171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:60947 (local address 127.9.117.195:34073)
1023 20:16:35.893325 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:35.893341 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:35.893358 (+ 17us) server_negotiation.cc:408] Connection header received
1023 20:16:35.893801 (+ 443us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:35.893804 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:35.893843 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:35.893977 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:35.895030 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:35.896236 (+ 1206us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:35.896912 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:35.897181 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:35.899878 (+ 2697us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:35.899891 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:35.899891 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:35.899923 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:35.902100 (+ 2177us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:35.902709 (+ 609us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:35.902713 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:35.902713 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:35.902772 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:35.902981 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:35.902982 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:35.902982 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:35.903218 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:35.903311 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:35.903586 (+ 275us) server_negotiation.cc:300] Negotiation successful
1023 20:16:35.903711 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20251023 20:16:35.903970 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:35.892939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:46123 (local address 127.9.117.193:49539)
1023 20:16:35.893003 (+ 64us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:35.893017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:35.893101 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:35.894033 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:35.894036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:35.894560 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:35.894892 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:35.894898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:35.896370 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:35.896372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:35.896841 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:35.896843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:35.896966 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:35.897818 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:35.897856 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:35.900052 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:35.902280 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:35.902282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:35.902282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:35.902616 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:35.902836 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:35.902837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:35.902838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:35.902920 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:35.903288 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:35.903294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:35.903421 (+ 127us) client_negotiation.cc:769] Sending connection context
1023 20:16:35.903524 (+ 103us) client_negotiation.cc:240] Negotiation successful
1023 20:16:35.903807 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":127}
W20251023 20:16:35.905017 27003 leader_election.cc:343] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073): Not found: Tablet not found: d7d9b57df86e4200a956ef71a07a385f
W20251023 20:16:35.905124 27005 leader_election.cc:343] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123): Not found: Tablet not found: d7d9b57df86e4200a956ef71a07a385f
I20251023 20:16:35.905205 27005 leader_election.cc:304] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49f86bf31f6e435e94e1b0265114c561; no voters: 0285ec2f90e64172a0d90a77d2725b4e, 4eec434ee63c4c4e8563fb9e7ca707f2
I20251023 20:16:35.905325 27390 raft_consensus.cc:2749] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:16:36.476821 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:36.476466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:46797)
1023 20:16:36.476651 (+ 185us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:36.476718 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20251023 20:16:36.525674 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:36.525430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:41725)
1023 20:16:36.525577 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:36.525622 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20251023 20:16:36.649267 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:36.649044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:53863)
1023 20:16:36.649170 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:36.649215 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20251023 20:16:37.477684 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:37.477422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:34429)
1023 20:16:37.477586 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:37.477625 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20251023 20:16:37.526427 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:37.526200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:41087)
1023 20:16:37.526334 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:37.526377 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20251023 20:16:37.650161 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:37.649819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:60841)
1023 20:16:37.650042 (+ 223us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:37.650091 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20251023 20:16:38.050004 27400 raft_consensus.cc:493] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:38.050123 27400 raft_consensus.cc:515] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:38.050352 27400 leader_election.cc:290] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073), 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
W20251023 20:16:38.050814 27003 leader_election.cc:343] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073): Not found: Tablet not found: d7d9b57df86e4200a956ef71a07a385f
W20251023 20:16:38.050891 27005 leader_election.cc:343] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123): Not found: Tablet not found: d7d9b57df86e4200a956ef71a07a385f
I20251023 20:16:38.050962 27005 leader_election.cc:304] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49f86bf31f6e435e94e1b0265114c561; no voters: 0285ec2f90e64172a0d90a77d2725b4e, 4eec434ee63c4c4e8563fb9e7ca707f2
I20251023 20:16:38.051054 27400 raft_consensus.cc:2749] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:16:38.478619 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:38.478320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:41023)
1023 20:16:38.478489 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:38.478538 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20251023 20:16:38.527179 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:38.526959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:43299)
1023 20:16:38.527085 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:38.527128 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20251023 20:16:38.651018 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:38.650722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:52899)
1023 20:16:38.650882 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:38.650947 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20251023 20:16:39.479485 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:39.479228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:33975)
1023 20:16:39.479390 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:39.479432 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20251023 20:16:39.527951 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:39.527697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:54683)
1023 20:16:39.527853 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:39.527897 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20251023 20:16:39.651762 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:39.651517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:33125)
1023 20:16:39.651655 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:39.651709 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20251023 20:16:40.480243 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:40.479972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:46035)
1023 20:16:40.480132 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:40.480180 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20251023 20:16:40.528854 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:40.528575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:56711)
1023 20:16:40.528715 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:40.528797 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20251023 20:16:40.652582 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:40.652335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:36939)
1023 20:16:40.652477 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:40.652527 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20251023 20:16:40.836120 27410 raft_consensus.cc:493] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:40.836237 27410 raft_consensus.cc:515] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:40.836402 27410 leader_election.cc:290] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073), 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
W20251023 20:16:40.836912 27005 leader_election.cc:343] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123): Not found: Tablet not found: d7d9b57df86e4200a956ef71a07a385f
W20251023 20:16:40.837002 27003 leader_election.cc:343] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073): Not found: Tablet not found: d7d9b57df86e4200a956ef71a07a385f
I20251023 20:16:40.837046 27003 leader_election.cc:304] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49f86bf31f6e435e94e1b0265114c561; no voters: 0285ec2f90e64172a0d90a77d2725b4e, 4eec434ee63c4c4e8563fb9e7ca707f2
I20251023 20:16:40.837127 27410 raft_consensus.cc:2749] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:16:41.481070 27411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:41.480832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:48435)
1023 20:16:41.480967 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:41.481011 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20251023 20:16:41.529661 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:41.529380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:49331)
1023 20:16:41.529544 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:41.529599 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20251023 20:16:41.653478 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:41.653207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:33847)
1023 20:16:41.653362 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:41.653410 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20251023 20:16:42.481912 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:42.481652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:41209)
1023 20:16:42.481811 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:42.481858 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20251023 20:16:42.530519 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:42.530218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:44099)
1023 20:16:42.530409 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:42.530456 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20251023 20:16:42.654345 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:42.654076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:43481)
1023 20:16:42.654244 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:42.654291 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20251023 20:16:42.838220 27417 raft_consensus.cc:493] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:42.838344 27417 raft_consensus.cc:515] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec434ee63c4c4e8563fb9e7ca707f2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34073 } } peers { permanent_uuid: "0285ec2f90e64172a0d90a77d2725b4e" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 46123 } } peers { permanent_uuid: "49f86bf31f6e435e94e1b0265114c561" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45485 } }
I20251023 20:16:42.838555 27417 leader_election.cc:290] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073), 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123)
W20251023 20:16:42.839003 27003 leader_election.cc:343] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eec434ee63c4c4e8563fb9e7ca707f2 (127.9.117.195:34073): Not found: Tablet not found: d7d9b57df86e4200a956ef71a07a385f
W20251023 20:16:42.839092 27005 leader_election.cc:343] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0285ec2f90e64172a0d90a77d2725b4e (127.9.117.194:46123): Not found: Tablet not found: d7d9b57df86e4200a956ef71a07a385f
I20251023 20:16:42.839135 27005 leader_election.cc:304] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49f86bf31f6e435e94e1b0265114c561; no voters: 0285ec2f90e64172a0d90a77d2725b4e, 4eec434ee63c4c4e8563fb9e7ca707f2
I20251023 20:16:42.839257 27417 raft_consensus.cc:2749] T d7d9b57df86e4200a956ef71a07a385f P 49f86bf31f6e435e94e1b0265114c561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:16:43.482833 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:43.482539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:54857)
1023 20:16:43.482713 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:43.482767 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
I20251023 20:16:43.531348 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:43.531051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:54293)
1023 20:16:43.531241 (+ 190us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:43.531289 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20251023 20:16:43.655138 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:43.654877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:46983)
1023 20:16:43.655044 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:43.655083 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20251023 20:16:44.483724 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:44.483455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:60439)
1023 20:16:44.483617 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:44.483667 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20251023 20:16:44.532174 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:44.531923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:47359)
1023 20:16:44.532073 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:44.532119 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20251023 20:16:44.655905 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:44.655666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:56247)
1023 20:16:44.655816 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:44.655850 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20251023 20:16:45.484581 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:45.484307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:37285)
1023 20:16:45.484471 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:45.484526 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20251023 20:16:45.533032 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:45.532792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:55525)
1023 20:16:45.532932 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:45.532977 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20251023 20:16:45.656630 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:45.656404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:59821)
1023 20:16:45.656533 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:45.656571 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20251023 20:16:46.485543 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:46.485181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:33525)
1023 20:16:46.485409 (+ 228us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:46.485461 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
I20251023 20:16:46.533874 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:46.533547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:51253)
1023 20:16:46.533764 (+ 217us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:46.533817 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20251023 20:16:46.657325 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:46.657086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:48905)
1023 20:16:46.657228 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:46.657271 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20251023 20:16:47.486416 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:47.486099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:34173)
1023 20:16:47.486310 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:47.486356 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
I20251023 20:16:47.534686 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:47.534417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:33645)
1023 20:16:47.534584 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:47.534630 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20251023 20:16:47.658038 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:47.657800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:34725)
1023 20:16:47.657943 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:47.657984 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20251023 20:16:48.487293 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:48.487018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:55673)
1023 20:16:48.487187 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:48.487234 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20251023 20:16:48.535600 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:48.535308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:50111)
1023 20:16:48.535485 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:48.535526 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20251023 20:16:48.658919 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:48.658640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:44291)
1023 20:16:48.658805 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:48.658857 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20251023 20:16:49.488260 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:49.487970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.193:53657)
1023 20:16:49.488156 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:49.488206 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20251023 20:16:49.536468 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:49.536217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.194:58009)
1023 20:16:49.536370 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:49.536415 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20251023 20:16:49.650048 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 26953
I20251023 20:16:49.655346 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 27120
I20251023 20:16:49.659821 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:49.659536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37775 (local address 127.9.117.195:57443)
1023 20:16:49.659713 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:49.659768 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.254:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
I20251023 20:16:49.660285 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 27254
2025-10-23T20:16:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59702 ms)
[----------] 31 tests from SecurityITest (309542 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:16:49 dist-test-slave-x62p krb5kdc[27441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:16:49 dist-test-slave-x62p krb5kdc[27441](info): set up 2 sockets
Oct 23 20:16:49 dist-test-slave-x62p krb5kdc[27441](info): commencing operation
krb5kdc: starting...
W20251023 20:16:51.665079 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.984s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:16:51 dist-test-slave-x62p krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:16:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:16:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:51.821518 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:45039
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:45039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:51.908206 27457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:51.908432 27457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:51.908457 27457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:51.910285 27457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:16:51.910357 27457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:51.910382 27457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:51.910449 27457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:16:51.910478 27457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:16:51.912744 27457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:45039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:45039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:51.913059 27457 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:51.913265 27457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:51.916174 27463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:51.916175 27465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:51.916174 27462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:51.916354 27457 server_base.cc:1047] running on GCE node
I20251023 20:16:51.916719 27457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:51.917034 27457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:51.918192 27457 hybrid_clock.cc:648] HybridClock initialized: now 1761250611918174 us; error 30 us; skew 500 ppm
Oct 23 20:16:51 dist-test-slave-x62p krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250611, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:51.921046 27457 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:16:51.921689 27457 webserver.cc:492] Webserver started at http://127.9.117.254:37051/ using document root <none> and password file <none>
I20251023 20:16:51.921880 27457 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:51.921914 27457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:51.922017 27457 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:51.923066 27457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "785634683d2f4652b9f185f9ccc1d9fb"
format_stamp: "Formatted at 2025-10-23 20:16:51 on dist-test-slave-x62p"
server_key: "f436639cccc679bfc0658d478632b5c1"
server_key_iv: "9c4f2ecfb092b48bcc47732c79477fbd"
server_key_version: "encryptionkey@0"
I20251023 20:16:51.923357 27457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "785634683d2f4652b9f185f9ccc1d9fb"
format_stamp: "Formatted at 2025-10-23 20:16:51 on dist-test-slave-x62p"
server_key: "f436639cccc679bfc0658d478632b5c1"
server_key_iv: "9c4f2ecfb092b48bcc47732c79477fbd"
server_key_version: "encryptionkey@0"
I20251023 20:16:51.924682 27457 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:16:51.925671 27472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:51.925933 27457 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:16:51.926008 27457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "785634683d2f4652b9f185f9ccc1d9fb"
format_stamp: "Formatted at 2025-10-23 20:16:51 on dist-test-slave-x62p"
server_key: "f436639cccc679bfc0658d478632b5c1"
server_key_iv: "9c4f2ecfb092b48bcc47732c79477fbd"
server_key_version: "encryptionkey@0"
I20251023 20:16:51.926074 27457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:51.946720 27457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:51.949231 27457 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:51.949381 27457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:51.954145 27457 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:45039
I20251023 20:16:51.954195 27524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:45039 every 8 connection(s)
I20251023 20:16:51.954592 27457 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:16:51.955297 27525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:51.957048 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 27457
I20251023 20:16:51.957196 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:16:51.957394 9687 external_mini_cluster.cc:1468] Setting key de1c49b6e6ec5395ea4fa76dac189feb
I20251023 20:16:51.958842 27525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb: Bootstrap starting.
I20251023 20:16:51.959515 27525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:51.959846 27525 log.cc:826] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:51.960713 27525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb: No bootstrap required, opened a new log
I20251023 20:16:51.962163 27525 raft_consensus.cc:359] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785634683d2f4652b9f185f9ccc1d9fb" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45039 } }
I20251023 20:16:51.962311 27525 raft_consensus.cc:385] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:51.962332 27525 raft_consensus.cc:740] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785634683d2f4652b9f185f9ccc1d9fb, State: Initialized, Role: FOLLOWER
I20251023 20:16:51.962435 27525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785634683d2f4652b9f185f9ccc1d9fb" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45039 } }
I20251023 20:16:51.962550 27525 raft_consensus.cc:399] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:16:51.962607 27525 raft_consensus.cc:493] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:16:51.962653 27525 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:51.963464 27525 raft_consensus.cc:515] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785634683d2f4652b9f185f9ccc1d9fb" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45039 } }
I20251023 20:16:51.963617 27525 leader_election.cc:304] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 785634683d2f4652b9f185f9ccc1d9fb; no voters:
Oct 23 20:16:51 dist-test-slave-x62p krb5kdc[27441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:51.963984 27525 leader_election.cc:290] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:16:51.964056 27530 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:51.964210 27530 raft_consensus.cc:697] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [term 1 LEADER]: Becoming Leader. State: Replica: 785634683d2f4652b9f185f9ccc1d9fb, State: Running, Role: LEADER
I20251023 20:16:51.964318 27525 sys_catalog.cc:565] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:16:51.964363 27530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785634683d2f4652b9f185f9ccc1d9fb" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45039 } }
I20251023 20:16:51.964818 27530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 785634683d2f4652b9f185f9ccc1d9fb. Latest consensus state: current_term: 1 leader_uuid: "785634683d2f4652b9f185f9ccc1d9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785634683d2f4652b9f185f9ccc1d9fb" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45039 } } }
I20251023 20:16:51.964918 27530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:51.965077 27531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "785634683d2f4652b9f185f9ccc1d9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785634683d2f4652b9f185f9ccc1d9fb" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45039 } } }
I20251023 20:16:51.965227 27531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:51.965397 27534 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:16:51.966123 27534 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:16:51.968137 27534 catalog_manager.cc:1357] Generated new cluster ID: 4a14472ca1234daf96da6ab7a826704f
I20251023 20:16:51.968209 27534 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:16:51.969657 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:51.958045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.9.117.254:45039)
1023 20:16:51.958221 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:51.958234 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:51.958254 (+ 20us) server_negotiation.cc:408] Connection header received
1023 20:16:51.958615 (+ 361us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:51.958621 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:51.958671 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:51.958831 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:51.959733 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:51.960817 (+ 1084us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:51.961484 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:51.961760 (+ 276us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:51.964459 (+ 2699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:51.964470 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:51.964471 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:51.964506 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:51.967717 (+ 3211us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:51.968327 (+ 610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:51.968331 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:51.968331 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:51.968412 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:51.968651 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:51.968652 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:51.968653 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:51.968971 (+ 318us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:51.969118 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:51.969320 (+ 202us) server_negotiation.cc:300] Negotiation successful
1023 20:16:51.969466 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20251023 20:16:51.978807 27534 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:16:51.979583 27534 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:16:51.986050 27534 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb: Generated new TSK 0
I20251023 20:16:51.986294 27534 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:52.044152 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:45039
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:52.134474 27553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:52.134680 27553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:52.134707 27553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:52.136390 27553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:52.136464 27553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:52.136497 27553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:16:52.138437 27553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:45039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:52.138754 27553 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:52.138991 27553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:52.141815 27558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:52.141834 27561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:52.141816 27559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:52.142043 27553 server_base.cc:1047] running on GCE node
I20251023 20:16:52.142330 27553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:52.142575 27553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:52.143750 27553 hybrid_clock.cc:648] HybridClock initialized: now 1761250612143730 us; error 37 us; skew 500 ppm
Oct 23 20:16:52 dist-test-slave-x62p krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250612, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:52.146823 27553 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:16:52.147467 27553 webserver.cc:492] Webserver started at http://127.9.117.193:33133/ using document root <none> and password file <none>
I20251023 20:16:52.147681 27553 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:52.147727 27553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:52.147852 27553 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:52.149029 27553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "20142fef6942499d87fa4eedea9e866e"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "1015a5d0a02807ba5a1a77ca71310186"
server_key_iv: "3c9b0dd99e77e42d98164ac2ff1f3db4"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.149482 27553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "20142fef6942499d87fa4eedea9e866e"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "1015a5d0a02807ba5a1a77ca71310186"
server_key_iv: "3c9b0dd99e77e42d98164ac2ff1f3db4"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.151860 27553 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20251023 20:16:52.152871 27568 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.153165 27553 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:16:52.153271 27553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "20142fef6942499d87fa4eedea9e866e"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "1015a5d0a02807ba5a1a77ca71310186"
server_key_iv: "3c9b0dd99e77e42d98164ac2ff1f3db4"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.153354 27553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:52.167018 27553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:52.169497 27553 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:52.169668 27553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:52.169931 27553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:52.170286 27553 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:52.170321 27553 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.170360 27553 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:52.170392 27553 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.177434 27553 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:42333
I20251023 20:16:52.177496 27681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:42333 every 8 connection(s)
I20251023 20:16:52.177826 27553 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:16:52.179472 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 27553
I20251023 20:16:52.179598 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:16:52.179826 9687 external_mini_cluster.cc:1468] Setting key 3a3f8ffa8a022d9070305de05b1b2bac
Oct 23 20:16:52 dist-test-slave-x62p krb5kdc[27441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250612, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:16:52.188604 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.178666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:33503 (local address 127.9.117.254:45039)
1023 20:16:52.178771 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.178773 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.178837 (+ 64us) server_negotiation.cc:408] Connection header received
1023 20:16:52.179424 (+ 587us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.179426 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.179443 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.179560 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:52.180684 (+ 1124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.181594 (+ 910us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.183027 (+ 1433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.183228 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.185750 (+ 2522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:52.185755 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:52.185755 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:52.185764 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:52.187264 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:52.187701 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:52.187702 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:52.187703 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:52.187732 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:52.187920 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:52.187921 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:52.187921 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:52.188034 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:52.188098 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.188369 (+ 271us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.188498 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251023 20:16:52.188776 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.178559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:45039 (local address 127.9.117.193:33503)
1023 20:16:52.178721 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.178754 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:52.179304 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:52.179632 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:52.179637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:52.180121 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:52.180561 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.180567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.181720 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.181722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:52.182906 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.182910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.183108 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.183716 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:52.183754 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:52.185632 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:52.187366 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:52.187375 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:52.187379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:52.187625 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:52.187791 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:52.187792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:52.187793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:52.187865 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:52.188091 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:52.188096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:52.188241 (+ 145us) client_negotiation.cc:769] Sending connection context
1023 20:16:52.188345 (+ 104us) client_negotiation.cc:240] Negotiation successful
1023 20:16:52.188506 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20251023 20:16:52.189406 27682 heartbeater.cc:344] Connected to a master server at 127.9.117.254:45039
I20251023 20:16:52.189515 27682 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:52.189725 27682 heartbeater.cc:507] Master 127.9.117.254:45039 requested a full tablet report, sending...
I20251023 20:16:52.190302 27489 ts_manager.cc:194] Registered new tserver with Master: 20142fef6942499d87fa4eedea9e866e (127.9.117.193:42333)
I20251023 20:16:52.191022 27489 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:33503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:52.235911 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:45039
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:52.322093 27689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:52.322302 27689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:52.322348 27689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:52.323853 27689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:52.323907 27689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:52.323953 27689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:16:52.325649 27689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:45039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:52.325939 27689 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:52.326144 27689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:52.329108 27695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:52.329087 27694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:52.329078 27697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:52.329244 27689 server_base.cc:1047] running on GCE node
I20251023 20:16:52.329442 27689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:52.329664 27689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:52.330821 27689 hybrid_clock.cc:648] HybridClock initialized: now 1761250612330801 us; error 29 us; skew 500 ppm
Oct 23 20:16:52 dist-test-slave-x62p krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250612, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:52.333338 27689 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:16:52.333953 27689 webserver.cc:492] Webserver started at http://127.9.117.194:39347/ using document root <none> and password file <none>
I20251023 20:16:52.334144 27689 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:52.334185 27689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:52.334304 27689 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:52.335330 27689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "5976d3645500440d9f3a8fadd20ef6fa"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "2fc5bd7cb05b8a8dcf8349237e312601"
server_key_iv: "d44145a3a85776413937ed4078449e49"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.335642 27689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "5976d3645500440d9f3a8fadd20ef6fa"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "2fc5bd7cb05b8a8dcf8349237e312601"
server_key_iv: "d44145a3a85776413937ed4078449e49"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.336959 27689 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251023 20:16:52.337643 27704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.337818 27689 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.337893 27689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "5976d3645500440d9f3a8fadd20ef6fa"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "2fc5bd7cb05b8a8dcf8349237e312601"
server_key_iv: "d44145a3a85776413937ed4078449e49"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.337955 27689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:52.353809 27689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:52.356386 27689 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:52.356710 27689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:52.357048 27689 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:52.357430 27689 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:52.357473 27689 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.357509 27689 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:52.357546 27689 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.364110 27689 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:33119
I20251023 20:16:52.364161 27817 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:33119 every 8 connection(s)
I20251023 20:16:52.364478 27689 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:16:52 dist-test-slave-x62p krb5kdc[27441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250612, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:52.371117 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 27689
I20251023 20:16:52.371237 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:16:52.371424 9687 external_mini_cluster.cc:1468] Setting key 05ef97569a71a0a7e5a96309541b0c2b
I20251023 20:16:52.374251 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.365277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:48083 (local address 127.9.117.254:45039)
1023 20:16:52.365337 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.365340 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.365528 (+ 188us) server_negotiation.cc:408] Connection header received
1023 20:16:52.366052 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.366055 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.366071 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.366155 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:52.367015 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.367527 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.368664 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.368867 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.370950 (+ 2083us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:52.370955 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:52.370956 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:52.370964 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:52.372842 (+ 1878us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:52.373249 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:52.373250 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:52.373251 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:52.373281 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:52.373465 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:52.373466 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:52.373466 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:52.373615 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:52.373698 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.374025 (+ 327us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.374146 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251023 20:16:52.374361 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.365247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:45039 (local address 127.9.117.194:48083)
1023 20:16:52.365388 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.365420 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:52.365937 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:52.366215 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:52.366219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:52.366571 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:52.366889 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.366892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.367644 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.367645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:52.368551 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.368553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.368679 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.369280 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:52.369301 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:52.370834 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:52.372930 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:52.372939 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:52.372943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:52.373173 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:52.373338 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:52.373339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:52.373339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:52.373407 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:52.373707 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:52.373716 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:52.373887 (+ 171us) client_negotiation.cc:769] Sending connection context
1023 20:16:52.373994 (+ 107us) client_negotiation.cc:240] Negotiation successful
1023 20:16:52.374124 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20251023 20:16:52.375164 27818 heartbeater.cc:344] Connected to a master server at 127.9.117.254:45039
I20251023 20:16:52.375277 27818 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:52.375629 27818 heartbeater.cc:507] Master 127.9.117.254:45039 requested a full tablet report, sending...
I20251023 20:16:52.376121 27489 ts_manager.cc:194] Registered new tserver with Master: 5976d3645500440d9f3a8fadd20ef6fa (127.9.117.194:33119)
I20251023 20:16:52.376514 27489 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:48083
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:52.421855 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:45039
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:52.507012 27825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:52.507202 27825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:52.507246 27825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:52.508796 27825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:52.508874 27825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:52.508919 27825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:16:52.510653 27825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:45039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:52.510963 27825 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:52.511173 27825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:52.513867 27833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:52.513978 27831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:52.514031 27830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:52.514241 27825 server_base.cc:1047] running on GCE node
I20251023 20:16:52.514405 27825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:52.514593 27825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:52.515746 27825 hybrid_clock.cc:648] HybridClock initialized: now 1761250612515722 us; error 38 us; skew 500 ppm
Oct 23 20:16:52 dist-test-slave-x62p krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250612, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:52.518412 27825 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:16:52.519030 27825 webserver.cc:492] Webserver started at http://127.9.117.195:38371/ using document root <none> and password file <none>
I20251023 20:16:52.519225 27825 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:52.519263 27825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:52.519369 27825 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:52.520427 27825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "568a0f426eb644a7a3f77c454ddb6fa2"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "bca194990895f929a5f4c57d7c885f7e"
server_key_iv: "1acae13fe95f4e6b707390e1c38664ff"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.520803 27825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "568a0f426eb644a7a3f77c454ddb6fa2"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "bca194990895f929a5f4c57d7c885f7e"
server_key_iv: "1acae13fe95f4e6b707390e1c38664ff"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.522110 27825 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:52.522930 27840 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.523139 27825 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:52.523226 27825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "568a0f426eb644a7a3f77c454ddb6fa2"
format_stamp: "Formatted at 2025-10-23 20:16:52 on dist-test-slave-x62p"
server_key: "bca194990895f929a5f4c57d7c885f7e"
server_key_iv: "1acae13fe95f4e6b707390e1c38664ff"
server_key_version: "encryptionkey@0"
I20251023 20:16:52.523294 27825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:52.535657 27825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:52.537895 27825 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:52.538056 27825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:52.538312 27825 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:52.538652 27825 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:52.538689 27825 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.538726 27825 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:52.538753 27825 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:52.545248 27825 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:35275
I20251023 20:16:52.545286 27953 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:35275 every 8 connection(s)
I20251023 20:16:52.545631 27825 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:16:52.546666 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 27825
I20251023 20:16:52.546756 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:16:52.546929 9687 external_mini_cluster.cc:1468] Setting key 968bbeb322bfd3038fdeef5756a27554
Oct 23 20:16:52 dist-test-slave-x62p krb5kdc[27441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250612, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:52.554553 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.546393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:34361 (local address 127.9.117.254:45039)
1023 20:16:52.546460 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.546463 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.546664 (+ 201us) server_negotiation.cc:408] Connection header received
1023 20:16:52.547314 (+ 650us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.547315 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.547329 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.547384 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:52.548321 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.548881 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.549930 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.550069 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.552088 (+ 2019us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:52.552092 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:52.552092 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:52.552099 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:52.553495 (+ 1396us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:52.553862 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:52.553863 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:52.553863 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:52.553887 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:52.554038 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:52.554038 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:52.554038 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:52.554130 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:52.554179 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.554395 (+ 216us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.554472 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251023 20:16:52.554682 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.546338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:45039 (local address 127.9.117.195:34361)
1023 20:16:52.546520 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.546582 (+ 62us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:52.547194 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:52.547463 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:52.547468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:52.547926 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:52.548239 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.548243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.548962 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.548963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:52.549851 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.549853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.549981 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.550545 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:52.550561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:52.552005 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:52.553582 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:52.553589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:52.553591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:52.553789 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:52.553940 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:52.553941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:52.553941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:52.553994 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:52.554171 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:52.554174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:52.554293 (+ 119us) client_negotiation.cc:769] Sending connection context
1023 20:16:52.554379 (+ 86us) client_negotiation.cc:240] Negotiation successful
1023 20:16:52.554490 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20251023 20:16:52.555225 27954 heartbeater.cc:344] Connected to a master server at 127.9.117.254:45039
I20251023 20:16:52.555326 27954 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:52.555527 27954 heartbeater.cc:507] Master 127.9.117.254:45039 requested a full tablet report, sending...
I20251023 20:16:52.555974 27489 ts_manager.cc:194] Registered new tserver with Master: 568a0f426eb644a7a3f77c454ddb6fa2 (127.9.117.195:35275)
I20251023 20:16:52.556362 27489 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:34361
I20251023 20:16:52.559072 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:16:52.566642 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.560586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.9.117.254:45039)
1023 20:16:52.560657 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.560660 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.560822 (+ 162us) server_negotiation.cc:408] Connection header received
1023 20:16:52.560870 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.560871 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.560882 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.560938 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:52.561660 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.562189 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.562760 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.562897 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.563954 (+ 1057us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:52.563959 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:52.563959 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:52.563966 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:52.565439 (+ 1473us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:52.565822 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:52.565824 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:52.565824 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:52.565852 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:52.566000 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:52.566001 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:52.566001 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:52.566135 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:52.566223 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.566369 (+ 146us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.566515 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251023 20:16:52.568498 27489 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:16:52.569994 27489 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:16:52.576153 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.572448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:42333 (local address 127.0.0.1:37074)
1023 20:16:52.573187 (+ 739us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.573196 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:52.573239 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:52.573783 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:52.573784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:52.573787 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:52.574003 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.574004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.575142 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.575143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:52.575894 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.575897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.575968 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.575971 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:52.576004 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:16:52.576045 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":9,"thread_start_us":111,"threads_started":1}
I20251023 20:16:52.576416 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.572444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:35275 (local address 127.0.0.1:43042)
1023 20:16:52.572794 (+ 350us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.572826 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:52.572931 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:52.573203 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:52.573207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:52.573210 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:52.573533 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.573536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.574776 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.574777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:52.575765 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.575768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.575853 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.575901 (+ 48us) client_negotiation.cc:769] Sending connection context
1023 20:16:52.575956 (+ 55us) client_negotiation.cc:240] Negotiation successful
1023 20:16:52.576003 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":125,"threads_started":1}
I20251023 20:16:52.576153 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.572448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:33119 (local address 127.0.0.1:33798)
1023 20:16:52.573045 (+ 597us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.573055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:52.573113 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:52.573445 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:52.573446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:52.573448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:52.573718 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.573720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.574731 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.574732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:52.575775 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.575776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.575851 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.575901 (+ 50us) client_negotiation.cc:769] Sending connection context
1023 20:16:52.575956 (+ 55us) client_negotiation.cc:240] Negotiation successful
1023 20:16:52.576003 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":111,"threads_started":1}
I20251023 20:16:52.576975 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.572775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.9.117.193:42333)
1023 20:16:52.573595 (+ 820us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.573604 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.573614 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:52.573660 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.573663 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.573683 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.573756 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:52.574083 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.575062 (+ 979us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.576129 (+ 1067us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.576695 (+ 566us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.576795 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.576829 (+ 34us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.576879 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":108,"threads_started":1}
I20251023 20:16:52.577361 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.572852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.9.117.194:33119)
1023 20:16:52.573006 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.573014 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.573161 (+ 147us) server_negotiation.cc:408] Connection header received
1023 20:16:52.573224 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.573227 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.573251 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.573335 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:52.573810 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.574650 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.576549 (+ 1899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.577159 (+ 610us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.577211 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.577243 (+ 32us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.577287 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20251023 20:16:52.577657 27616 tablet_service.cc:1505] Processing CreateTablet for tablet 4dd32fe580654fc8b2fd10e889fa46f2 (DEFAULT_TABLE table=test-table [id=5afb0f83b5414a72b21cfa27005bc2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:52.577909 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.572634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.9.117.195:35275)
1023 20:16:52.572817 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.572827 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.572920 (+ 93us) server_negotiation.cc:408] Connection header received
1023 20:16:52.573043 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.573046 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.573069 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.573157 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:52.573615 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.574705 (+ 1090us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.575885 (+ 1180us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.577691 (+ 1806us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.577729 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.577756 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.577799 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20251023 20:16:52.578018 27616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd32fe580654fc8b2fd10e889fa46f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:52.577971 27752 tablet_service.cc:1505] Processing CreateTablet for tablet 4dd32fe580654fc8b2fd10e889fa46f2 (DEFAULT_TABLE table=test-table [id=5afb0f83b5414a72b21cfa27005bc2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:52.578230 27752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd32fe580654fc8b2fd10e889fa46f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:52.581276 27972 tablet_bootstrap.cc:492] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Bootstrap starting.
I20251023 20:16:52.581607 27888 tablet_service.cc:1505] Processing CreateTablet for tablet 4dd32fe580654fc8b2fd10e889fa46f2 (DEFAULT_TABLE table=test-table [id=5afb0f83b5414a72b21cfa27005bc2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:52.581914 27888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd32fe580654fc8b2fd10e889fa46f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:52.582075 27972 tablet_bootstrap.cc:654] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:52.582381 27972 log.cc:826] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:52.582736 27971 tablet_bootstrap.cc:492] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e: Bootstrap starting.
I20251023 20:16:52.583201 27972 tablet_bootstrap.cc:492] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: No bootstrap required, opened a new log
I20251023 20:16:52.583273 27972 ts_tablet_manager.cc:1403] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:16:52.583462 27971 tablet_bootstrap.cc:654] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:52.583808 27971 log.cc:826] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:52.584472 27975 tablet_bootstrap.cc:492] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Bootstrap starting.
I20251023 20:16:52.584867 27971 tablet_bootstrap.cc:492] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e: No bootstrap required, opened a new log
I20251023 20:16:52.584995 27971 ts_tablet_manager.cc:1403] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:52.585219 27975 tablet_bootstrap.cc:654] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:52.585493 27975 log.cc:826] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:52.585493 27972 raft_consensus.cc:359] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.585659 27972 raft_consensus.cc:385] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:52.585727 27972 raft_consensus.cc:740] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5976d3645500440d9f3a8fadd20ef6fa, State: Initialized, Role: FOLLOWER
I20251023 20:16:52.585868 27972 consensus_queue.cc:260] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.586177 27972 ts_tablet_manager.cc:1434] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:16:52.586251 27818 heartbeater.cc:499] Master 127.9.117.254:45039 was elected leader, sending a full tablet report...
I20251023 20:16:52.586910 27975 tablet_bootstrap.cc:492] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: No bootstrap required, opened a new log
I20251023 20:16:52.586984 27975 ts_tablet_manager.cc:1403] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251023 20:16:52.587145 27971 raft_consensus.cc:359] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.587291 27971 raft_consensus.cc:385] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:52.587322 27971 raft_consensus.cc:740] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20142fef6942499d87fa4eedea9e866e, State: Initialized, Role: FOLLOWER
I20251023 20:16:52.587464 27971 consensus_queue.cc:260] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.587839 27971 ts_tablet_manager.cc:1434] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:16:52.587903 27682 heartbeater.cc:499] Master 127.9.117.254:45039 was elected leader, sending a full tablet report...
I20251023 20:16:52.589167 27975 raft_consensus.cc:359] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.589303 27975 raft_consensus.cc:385] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:52.589322 27975 raft_consensus.cc:740] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568a0f426eb644a7a3f77c454ddb6fa2, State: Initialized, Role: FOLLOWER
I20251023 20:16:52.589393 27975 consensus_queue.cc:260] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.589630 27975 ts_tablet_manager.cc:1434] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251023 20:16:52.589699 27954 heartbeater.cc:499] Master 127.9.117.254:45039 was elected leader, sending a full tablet report...
W20251023 20:16:52.615120 27819 tablet.cc:2378] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:16:52.678701 27683 tablet.cc:2378] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:16:52.796216 27955 tablet.cc:2378] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:16:52.805433 27977 raft_consensus.cc:493] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:52.805569 27977 raft_consensus.cc:515] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.805977 27977 leader_election.cc:290] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 568a0f426eb644a7a3f77c454ddb6fa2 (127.9.117.195:35275), 20142fef6942499d87fa4eedea9e866e (127.9.117.193:42333)
I20251023 20:16:52.808993 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.806163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:35275 (local address 127.9.117.194:57029)
1023 20:16:52.806236 (+ 73us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.806251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:52.806306 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:52.806467 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:52.806469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:52.806471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:52.806747 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.806749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.807712 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.807714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:52.808709 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.808712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.808820 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.808825 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:16:52.808862 (+ 37us) client_negotiation.cc:240] Negotiation successful
1023 20:16:52.808902 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251023 20:16:52.809001 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.806165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:42333 (local address 127.9.117.194:60237)
1023 20:16:52.806409 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.806416 (+ 7us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:52.806474 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:52.806610 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:52.806611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:52.806612 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:52.806891 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.806893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.807754 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.807754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:52.808709 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:52.808712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.808819 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.808825 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:16:52.808862 (+ 37us) client_negotiation.cc:240] Negotiation successful
1023 20:16:52.808902 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":20,"thread_start_us":107,"threads_started":1}
I20251023 20:16:52.809486 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.806277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:60237 (local address 127.9.117.193:42333)
1023 20:16:52.806339 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.806342 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.806484 (+ 142us) server_negotiation.cc:408] Connection header received
1023 20:16:52.806523 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.806524 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.806538 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.806592 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:52.806964 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.807661 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.808864 (+ 1203us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.809353 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.809363 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.809386 (+ 23us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.809419 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251023 20:16:52.809486 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.806252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:57029 (local address 127.9.117.195:35275)
1023 20:16:52.806310 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.806312 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.806349 (+ 37us) server_negotiation.cc:408] Connection header received
1023 20:16:52.806386 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.806387 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.806399 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.806443 (+ 44us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:52.806851 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.807618 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.808864 (+ 1246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.809357 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.809363 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.809387 (+ 24us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.809419 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251023 20:16:52.809862 27908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dd32fe580654fc8b2fd10e889fa46f2" candidate_uuid: "5976d3645500440d9f3a8fadd20ef6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" is_pre_election: true
I20251023 20:16:52.809863 27636 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dd32fe580654fc8b2fd10e889fa46f2" candidate_uuid: "5976d3645500440d9f3a8fadd20ef6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20142fef6942499d87fa4eedea9e866e" is_pre_election: true
I20251023 20:16:52.810029 27908 raft_consensus.cc:2468] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5976d3645500440d9f3a8fadd20ef6fa in term 0.
I20251023 20:16:52.810029 27636 raft_consensus.cc:2468] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5976d3645500440d9f3a8fadd20ef6fa in term 0.
I20251023 20:16:52.810286 27706 leader_election.cc:304] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20142fef6942499d87fa4eedea9e866e, 5976d3645500440d9f3a8fadd20ef6fa; no voters:
I20251023 20:16:52.810442 27977 raft_consensus.cc:2804] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:16:52.810518 27977 raft_consensus.cc:493] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:16:52.810547 27977 raft_consensus.cc:3060] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:52.811268 27977 raft_consensus.cc:515] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.811406 27977 leader_election.cc:290] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [CANDIDATE]: Term 1 election: Requested vote from peers 568a0f426eb644a7a3f77c454ddb6fa2 (127.9.117.195:35275), 20142fef6942499d87fa4eedea9e866e (127.9.117.193:42333)
I20251023 20:16:52.811589 27908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dd32fe580654fc8b2fd10e889fa46f2" candidate_uuid: "5976d3645500440d9f3a8fadd20ef6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568a0f426eb644a7a3f77c454ddb6fa2"
I20251023 20:16:52.811599 27636 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dd32fe580654fc8b2fd10e889fa46f2" candidate_uuid: "5976d3645500440d9f3a8fadd20ef6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20142fef6942499d87fa4eedea9e866e"
I20251023 20:16:52.811666 27908 raft_consensus.cc:3060] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:52.811686 27636 raft_consensus.cc:3060] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:52.812340 27908 raft_consensus.cc:2468] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5976d3645500440d9f3a8fadd20ef6fa in term 1.
I20251023 20:16:52.812467 27636 raft_consensus.cc:2468] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5976d3645500440d9f3a8fadd20ef6fa in term 1.
I20251023 20:16:52.812546 27705 leader_election.cc:304] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568a0f426eb644a7a3f77c454ddb6fa2, 5976d3645500440d9f3a8fadd20ef6fa; no voters:
I20251023 20:16:52.812641 27977 raft_consensus.cc:2804] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:52.812821 27977 raft_consensus.cc:697] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 1 LEADER]: Becoming Leader. State: Replica: 5976d3645500440d9f3a8fadd20ef6fa, State: Running, Role: LEADER
I20251023 20:16:52.813020 27977 consensus_queue.cc:237] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } }
I20251023 20:16:52.813808 27489 catalog_manager.cc:5649] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 5976d3645500440d9f3a8fadd20ef6fa (127.9.117.194). New cstate: current_term: 1 leader_uuid: "5976d3645500440d9f3a8fadd20ef6fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5976d3645500440d9f3a8fadd20ef6fa" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 33119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 } health_report { overall_health: UNKNOWN } } }
I20251023 20:16:52.868927 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.866751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.9.117.194:33119)
1023 20:16:52.866824 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:52.866826 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:52.866836 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:52.866889 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:52.866890 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:52.866915 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:52.866972 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:16:52.867243 (+ 271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.867726 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:52.868321 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:52.868480 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:52.868528 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:16:52.868634 (+ 106us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:16:52.868712 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:52.868811 (+ 99us) server_negotiation.cc:300] Negotiation successful
1023 20:16:52.868844 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251023 20:16:52.870673 27908 raft_consensus.cc:1275] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 1 FOLLOWER]: Refusing update from remote peer 5976d3645500440d9f3a8fadd20ef6fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:52.870697 27636 raft_consensus.cc:1275] T 4dd32fe580654fc8b2fd10e889fa46f2 P 20142fef6942499d87fa4eedea9e866e [term 1 FOLLOWER]: Refusing update from remote peer 5976d3645500440d9f3a8fadd20ef6fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:52.871035 27977 consensus_queue.cc:1048] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "568a0f426eb644a7a3f77c454ddb6fa2" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:52.871173 27981 consensus_queue.cc:1048] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "20142fef6942499d87fa4eedea9e866e" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:52.873495 27984 mvcc.cc:204] Tried to move back new op lower bound from 7214082510315704320 to 7214082510082613248. Current Snapshot: MvccSnapshot[applied={T|T < 7214082510315704320}]
I20251023 20:16:52.874444 27986 mvcc.cc:204] Tried to move back new op lower bound from 7214082510315704320 to 7214082510082613248. Current Snapshot: MvccSnapshot[applied={T|T < 7214082510315704320 or (T in {7214082510315704320})}]
I20251023 20:16:52.874577 27985 mvcc.cc:204] Tried to move back new op lower bound from 7214082510315704320 to 7214082510082613248. Current Snapshot: MvccSnapshot[applied={T|T < 7214082510315704320 or (T in {7214082510315704320})}]
I20251023 20:16:52.875736 27488 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42518:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:52.875841 27488 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42518:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:52.876582 27488 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 785634683d2f4652b9f185f9ccc1d9fb: Sending DeleteTablet for 3 replicas of tablet 4dd32fe580654fc8b2fd10e889fa46f2
I20251023 20:16:52.876969 27888 tablet_service.cc:1552] Processing DeleteTablet for tablet 4dd32fe580654fc8b2fd10e889fa46f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:52 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:43042
I20251023 20:16:52.876988 27616 tablet_service.cc:1552] Processing DeleteTablet for tablet 4dd32fe580654fc8b2fd10e889fa46f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:52 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:37074
I20251023 20:16:52.877249 27752 tablet_service.cc:1552] Processing DeleteTablet for tablet 4dd32fe580654fc8b2fd10e889fa46f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:52 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:33798
I20251023 20:16:52.877475 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 27553
I20251023 20:16:52.877570 27996 tablet_replica.cc:333] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: stopping tablet replica
I20251023 20:16:52.877653 27996 raft_consensus.cc:2243] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:16:52.877768 27996 raft_consensus.cc:2272] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:52.879060 27996 ts_tablet_manager.cc:1916] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:52.879969 27994 tablet_replica.cc:333] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: stopping tablet replica
I20251023 20:16:52.880155 27994 raft_consensus.cc:2243] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:52.880324 27994 raft_consensus.cc:2272] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:52.880892 27994 ts_tablet_manager.cc:1916] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:52.881431 27996 ts_tablet_manager.cc:1929] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:52.881567 27996 log.cc:1199] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/4dd32fe580654fc8b2fd10e889fa46f2
I20251023 20:16:52.881968 27996 ts_tablet_manager.cc:1950] T 4dd32fe580654fc8b2fd10e889fa46f2 P 5976d3645500440d9f3a8fadd20ef6fa: Deleting consensus metadata
I20251023 20:16:52.882527 27476 catalog_manager.cc:4985] TS 5976d3645500440d9f3a8fadd20ef6fa (127.9.117.194:33119): tablet 4dd32fe580654fc8b2fd10e889fa46f2 (table test-table [id=5afb0f83b5414a72b21cfa27005bc2fa]) successfully deleted
I20251023 20:16:52.882652 27994 ts_tablet_manager.cc:1929] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:52.882815 27994 log.cc:1199] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/4dd32fe580654fc8b2fd10e889fa46f2
I20251023 20:16:52.883137 27994 ts_tablet_manager.cc:1950] T 4dd32fe580654fc8b2fd10e889fa46f2 P 568a0f426eb644a7a3f77c454ddb6fa2: Deleting consensus metadata
I20251023 20:16:52.883601 27473 catalog_manager.cc:4985] TS 568a0f426eb644a7a3f77c454ddb6fa2 (127.9.117.195:35275): tablet 4dd32fe580654fc8b2fd10e889fa46f2 (table test-table [id=5afb0f83b5414a72b21cfa27005bc2fa]) successfully deleted
W20251023 20:16:52.884537 27474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:42333 (error 108)
I20251023 20:16:52.884858 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 27689
I20251023 20:16:52.885494 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:52.884986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:42333 (local address 127.0.0.1:37090)
1023 20:16:52.885424 (+ 438us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:52.885462 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:42333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410}
W20251023 20:16:52.885648 27474 catalog_manager.cc:4712] TS 20142fef6942499d87fa4eedea9e866e (127.9.117.193:42333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4dd32fe580654fc8b2fd10e889fa46f2: Network error: Client connection negotiation failed: client connection to 127.9.117.193:42333: connect: Connection refused (error 111)
I20251023 20:16:52.891189 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 27825
I20251023 20:16:52.896790 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 27457
2025-10-23T20:16:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3236 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2025-10-23T20:16:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:16:52Z Disabled control of system clock
I20251023 20:16:52.925349 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:38281
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251023 20:16:53.012043 28001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:53.012241 28001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:53.012264 28001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:53.013729 28001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:16:53.013783 28001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:53.013798 28001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:53.013813 28001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:16:53.013826 28001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:16:53.015339 28001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:38281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:53.015588 28001 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:53.015810 28001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:53.018656 28007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:53.018767 28009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:53.018682 28006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:53.018914 28001 server_base.cc:1047] running on GCE node
I20251023 20:16:53.019119 28001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:53.019385 28001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:53.020534 28001 hybrid_clock.cc:648] HybridClock initialized: now 1761250613020502 us; error 51 us; skew 500 ppm
I20251023 20:16:53.021735 28001 webserver.cc:492] Webserver started at http://127.9.117.254:38169/ using document root <none> and password file <none>
I20251023 20:16:53.021965 28001 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:53.022012 28001 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:53.022145 28001 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:53.023178 28001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "7eed2b3e25284b6fa6c5c078f7902efe"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "96831f23d01043613c0766835a76023c"
server_key_iv: "a983ca8364b379398afdc517c2a30006"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.023496 28001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "7eed2b3e25284b6fa6c5c078f7902efe"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "96831f23d01043613c0766835a76023c"
server_key_iv: "a983ca8364b379398afdc517c2a30006"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.024847 28001 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:16:53.025636 28015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.025837 28001 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:16:53.025908 28001 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "7eed2b3e25284b6fa6c5c078f7902efe"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "96831f23d01043613c0766835a76023c"
server_key_iv: "a983ca8364b379398afdc517c2a30006"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.025979 28001 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:53.041927 28001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:53.042292 28001 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:53.042416 28001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:53.046742 28001 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:38281
I20251023 20:16:53.046792 28067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:38281 every 8 connection(s)
I20251023 20:16:53.047135 28001 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:16:53.047857 28068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:53.049822 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 28001
I20251023 20:16:53.049942 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:16:53.050122 9687 external_mini_cluster.cc:1468] Setting key bca93509fa3a694b162d4ca9705c2816
I20251023 20:16:53.051106 28068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe: Bootstrap starting.
I20251023 20:16:53.051911 28068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:53.052389 28068 log.cc:826] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:53.053272 28068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe: No bootstrap required, opened a new log
I20251023 20:16:53.054778 28068 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eed2b3e25284b6fa6c5c078f7902efe" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38281 } }
I20251023 20:16:53.054970 28068 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:53.055019 28068 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eed2b3e25284b6fa6c5c078f7902efe, State: Initialized, Role: FOLLOWER
I20251023 20:16:53.055146 28068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eed2b3e25284b6fa6c5c078f7902efe" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38281 } }
I20251023 20:16:53.055230 28068 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:16:53.055265 28068 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:16:53.055328 28068 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:53.055577 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.050709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.9.117.254:38281)
1023 20:16:53.051002 (+ 293us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.051012 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.051037 (+ 25us) server_negotiation.cc:408] Connection header received
1023 20:16:53.051461 (+ 424us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.051473 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.051577 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.051734 (+ 157us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:53.052246 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.053396 (+ 1150us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.054036 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.054310 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.054680 (+ 370us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:53.054690 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:53.054691 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:53.054936 (+ 245us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:53.054977 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:53.054987 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:53.055070 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.055316 (+ 246us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.055383 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":129,"threads_started":1}
I20251023 20:16:53.056135 28068 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eed2b3e25284b6fa6c5c078f7902efe" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38281 } }
I20251023 20:16:53.056258 28068 leader_election.cc:304] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eed2b3e25284b6fa6c5c078f7902efe; no voters:
I20251023 20:16:53.056447 28068 leader_election.cc:290] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:16:53.056551 28073 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:53.056687 28068 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:16:53.056721 28073 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [term 1 LEADER]: Becoming Leader. State: Replica: 7eed2b3e25284b6fa6c5c078f7902efe, State: Running, Role: LEADER
I20251023 20:16:53.056901 28073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eed2b3e25284b6fa6c5c078f7902efe" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38281 } }
I20251023 20:16:53.057271 28075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eed2b3e25284b6fa6c5c078f7902efe. Latest consensus state: current_term: 1 leader_uuid: "7eed2b3e25284b6fa6c5c078f7902efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eed2b3e25284b6fa6c5c078f7902efe" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38281 } } }
I20251023 20:16:53.057421 28075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:53.057287 28074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eed2b3e25284b6fa6c5c078f7902efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eed2b3e25284b6fa6c5c078f7902efe" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38281 } } }
I20251023 20:16:53.057543 28074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:53.057885 28082 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:16:53.058362 28082 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:16:53.060302 28082 catalog_manager.cc:1357] Generated new cluster ID: 45af9d1173b04451a0e2347388995822
I20251023 20:16:53.060357 28082 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:16:53.077013 28082 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:16:53.077682 28082 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:16:53.083509 28082 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe: Generated new TSK 0
I20251023 20:16:53.083763 28082 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:16:53.089677 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38281
--builtin_ntp_servers=127.9.117.212:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251023 20:16:53.186228 28092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:53.186407 28092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:53.186446 28092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:53.188077 28092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:53.188131 28092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:53.188174 28092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:16:53.189816 28092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:53.190105 28092 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:53.190354 28092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:53.193041 28098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:53.193037 28097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:53.193310 28100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:53.193439 28092 server_base.cc:1047] running on GCE node
I20251023 20:16:53.193612 28092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:53.193845 28092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:53.194977 28092 hybrid_clock.cc:648] HybridClock initialized: now 1761250613194958 us; error 35 us; skew 500 ppm
I20251023 20:16:53.196543 28092 webserver.cc:492] Webserver started at http://127.9.117.193:34583/ using document root <none> and password file <none>
I20251023 20:16:53.196828 28092 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:53.196889 28092 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:53.197017 28092 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:53.198370 28092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "c5b3c16cba764b9f8fe77b972db0970a"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "5494f3bbb24f5ce81c59480602032599"
server_key_iv: "8c99c5476b6ee1383f9c5a37bd739fbf"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.198767 28092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "c5b3c16cba764b9f8fe77b972db0970a"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "5494f3bbb24f5ce81c59480602032599"
server_key_iv: "8c99c5476b6ee1383f9c5a37bd739fbf"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.200404 28092 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:16:53.201465 28106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.201799 28092 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:16:53.201896 28092 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "c5b3c16cba764b9f8fe77b972db0970a"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "5494f3bbb24f5ce81c59480602032599"
server_key_iv: "8c99c5476b6ee1383f9c5a37bd739fbf"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.202023 28092 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:53.218142 28092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:53.218609 28092 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:53.218771 28092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:53.219060 28092 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:53.219480 28092 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:53.219544 28092 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.219589 28092 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:53.219615 28092 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.227416 28092 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:33063
I20251023 20:16:53.227624 28219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:33063 every 8 connection(s)
I20251023 20:16:53.227913 28092 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:16:53.232771 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.228649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:33745 (local address 127.9.117.254:38281)
1023 20:16:53.228752 (+ 103us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.228755 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.229037 (+ 282us) server_negotiation.cc:408] Connection header received
1023 20:16:53.229708 (+ 671us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.229710 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.229723 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.229805 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:53.230398 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.230916 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.231705 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.231838 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.232481 (+ 643us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:53.232486 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:53.232486 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:53.232491 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:53.232504 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:53.232514 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:53.232559 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.232640 (+ 81us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.232671 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251023 20:16:53.232936 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.228625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:38281 (local address 127.9.117.193:33745)
1023 20:16:53.228899 (+ 274us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.228930 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:53.229607 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:53.229876 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:53.229881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:53.229891 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:53.230325 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.230328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.230997 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.230998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:53.231622 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.231624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.231766 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.232357 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:16:53.232357 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:53.232383 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:16:53.232405 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:53.232569 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:53.232573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:53.232575 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:16:53.232644 (+ 69us) client_negotiation.cc:240] Negotiation successful
1023 20:16:53.232748 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20251023 20:16:53.233590 28220 heartbeater.cc:344] Connected to a master server at 127.9.117.254:38281
I20251023 20:16:53.233676 28220 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:53.233927 28220 heartbeater.cc:507] Master 127.9.117.254:38281 requested a full tablet report, sending...
I20251023 20:16:53.234476 28032 ts_manager.cc:194] Registered new tserver with Master: c5b3c16cba764b9f8fe77b972db0970a (127.9.117.193:33063)
I20251023 20:16:53.235265 28032 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.193:33745
I20251023 20:16:53.236161 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 28092
I20251023 20:16:53.236279 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:16:53.236508 9687 external_mini_cluster.cc:1468] Setting key 7ebed991986576c23673622c28290fb3
I20251023 20:16:53.237609 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38281
--builtin_ntp_servers=127.9.117.212:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251023 20:16:53.344445 28223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:53.344637 28223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:53.344664 28223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:53.346235 28223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:53.346289 28223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:53.346318 28223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:16:53.347823 28223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:53.348091 28223 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:53.348354 28223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:53.351277 28228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:53.351334 28229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:53.351503 28223 server_base.cc:1047] running on GCE node
W20251023 20:16:53.351282 28231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:53.351801 28223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:53.352035 28223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:53.353201 28223 hybrid_clock.cc:648] HybridClock initialized: now 1761250613353185 us; error 28 us; skew 500 ppm
I20251023 20:16:53.354411 28223 webserver.cc:492] Webserver started at http://127.9.117.194:43711/ using document root <none> and password file <none>
I20251023 20:16:53.354645 28223 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:53.354694 28223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:53.354820 28223 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:53.355852 28223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "20ffb5d229b7417eb7728b9389146156"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "75285e71c150650789904e7ca365c21e"
server_key_iv: "201e52161282d5bf01fadd5bee15ce5e"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.356168 28223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "20ffb5d229b7417eb7728b9389146156"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "75285e71c150650789904e7ca365c21e"
server_key_iv: "201e52161282d5bf01fadd5bee15ce5e"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.357568 28223 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:16:53.358338 28237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.358587 28223 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:53.358665 28223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "20ffb5d229b7417eb7728b9389146156"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "75285e71c150650789904e7ca365c21e"
server_key_iv: "201e52161282d5bf01fadd5bee15ce5e"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.358738 28223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:53.380390 28223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:53.380781 28223 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:53.380913 28223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:53.381156 28223 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:53.381480 28223 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:53.381522 28223 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.381559 28223 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:53.381588 28223 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.387966 28223 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:38905
I20251023 20:16:53.388084 28350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:38905 every 8 connection(s)
I20251023 20:16:53.388366 28223 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:16:53.392904 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 28223
I20251023 20:16:53.392999 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.389192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:39639 (local address 127.9.117.254:38281)
1023 20:16:53.389239 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.389242 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.389441 (+ 199us) server_negotiation.cc:408] Connection header received
1023 20:16:53.390020 (+ 579us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.390022 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.390045 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.390119 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:53.390637 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.391155 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.391794 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.391988 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.392631 (+ 643us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:53.392636 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:53.392636 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:53.392663 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:53.392678 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:53.392681 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:53.392759 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.392872 (+ 113us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.392917 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251023 20:16:53.393023 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:16:53.393142 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.389145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:38281 (local address 127.9.117.194:39639)
1023 20:16:53.389337 (+ 192us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.389367 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:53.389904 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:53.390193 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:53.390198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:53.390208 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:53.390544 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.390547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.391249 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.391250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:53.391692 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.391693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.391851 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.392484 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:16:53.392484 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:53.392509 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:16:53.392530 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:53.392759 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:53.392765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:53.392768 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:53.392852 (+ 84us) client_negotiation.cc:240] Negotiation successful
1023 20:16:53.392945 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20251023 20:16:53.393235 9687 external_mini_cluster.cc:1468] Setting key 5f02745beb7a4f2da3ba6456894fe834
I20251023 20:16:53.393812 28351 heartbeater.cc:344] Connected to a master server at 127.9.117.254:38281
I20251023 20:16:53.393922 28351 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:53.394193 28351 heartbeater.cc:507] Master 127.9.117.254:38281 requested a full tablet report, sending...
I20251023 20:16:53.394467 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38281
--builtin_ntp_servers=127.9.117.212:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251023 20:16:53.394671 28031 ts_manager.cc:194] Registered new tserver with Master: 20ffb5d229b7417eb7728b9389146156 (127.9.117.194:38905)
I20251023 20:16:53.395251 28031 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.194:39639
W20251023 20:16:53.492910 28354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:53.493109 28354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:53.493135 28354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:53.494632 28354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:53.494693 28354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:53.494722 28354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:16:53.496206 28354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:53.496439 28354 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:53.496675 28354 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:53.499558 28362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:53.499615 28360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:53.499791 28359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:53.499673 28354 server_base.cc:1047] running on GCE node
I20251023 20:16:53.500100 28354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:53.500324 28354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:53.501471 28354 hybrid_clock.cc:648] HybridClock initialized: now 1761250613501460 us; error 33 us; skew 500 ppm
I20251023 20:16:53.502740 28354 webserver.cc:492] Webserver started at http://127.9.117.195:37325/ using document root <none> and password file <none>
I20251023 20:16:53.502957 28354 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:53.503005 28354 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:53.503135 28354 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:53.504132 28354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "aa249608290e4f84b04331e71bdb85f1"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "9db131b72844ec3893e29b702e2fa6a7"
server_key_iv: "234ddc686bafdc3fad547cdb066bd5f1"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.504451 28354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "aa249608290e4f84b04331e71bdb85f1"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "9db131b72844ec3893e29b702e2fa6a7"
server_key_iv: "234ddc686bafdc3fad547cdb066bd5f1"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.505798 28354 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:16:53.506492 28368 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.506644 28354 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:16:53.506718 28354 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "aa249608290e4f84b04331e71bdb85f1"
format_stamp: "Formatted at 2025-10-23 20:16:53 on dist-test-slave-x62p"
server_key: "9db131b72844ec3893e29b702e2fa6a7"
server_key_iv: "234ddc686bafdc3fad547cdb066bd5f1"
server_key_version: "encryptionkey@0"
I20251023 20:16:53.506788 28354 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:53.517832 28354 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:53.518208 28354 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:53.518347 28354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:53.518623 28354 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:53.519178 28354 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:53.519233 28354 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.519294 28354 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:53.519323 28354 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:53.526494 28354 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:45879
I20251023 20:16:53.526629 28481 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:45879 every 8 connection(s)
I20251023 20:16:53.526904 28354 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:16:53.530260 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 28354
I20251023 20:16:53.530390 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:16:53.530592 9687 external_mini_cluster.cc:1468] Setting key b79b1b9d026ec612b9c8b15a04058c8d
I20251023 20:16:53.531509 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.527676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:34937 (local address 127.9.117.254:38281)
1023 20:16:53.527726 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.527729 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.527951 (+ 222us) server_negotiation.cc:408] Connection header received
1023 20:16:53.528516 (+ 565us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.528517 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.528530 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.528595 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:53.529134 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.529654 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.530286 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.530464 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.531200 (+ 736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:53.531205 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:53.531205 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:53.531211 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:53.531226 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:53.531232 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:53.531281 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.531393 (+ 112us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.531430 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251023 20:16:53.531725 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.527640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:38281 (local address 127.9.117.195:34937)
1023 20:16:53.527847 (+ 207us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.527872 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:53.528420 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:53.528666 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:53.528670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:53.528679 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:53.529049 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.529052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.529755 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.529757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:53.530198 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.530200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.530347 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.531057 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:16:53.531058 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:53.531099 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:16:53.531125 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:53.531287 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:53.531291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:53.531295 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:16:53.531378 (+ 83us) client_negotiation.cc:240] Negotiation successful
1023 20:16:53.531478 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20251023 20:16:53.532214 28482 heartbeater.cc:344] Connected to a master server at 127.9.117.254:38281
I20251023 20:16:53.532313 28482 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:53.532513 28482 heartbeater.cc:507] Master 127.9.117.254:38281 requested a full tablet report, sending...
I20251023 20:16:53.532891 28031 ts_manager.cc:194] Registered new tserver with Master: aa249608290e4f84b04331e71bdb85f1 (127.9.117.195:45879)
I20251023 20:16:53.533275 28031 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.195:34937
I20251023 20:16:53.542789 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:16:53.547035 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.544347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.9.117.254:38281)
1023 20:16:53.544440 (+ 93us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.544442 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.544488 (+ 46us) server_negotiation.cc:408] Connection header received
1023 20:16:53.544552 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.544553 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.544564 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.544617 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:53.544965 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.545465 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.546051 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.546170 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.546747 (+ 577us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:53.546751 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:53.546752 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:53.546758 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:53.546774 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:53.546778 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:53.546829 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.546896 (+ 67us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.546929 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251023 20:16:53.548987 28031 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:16:53.550463 28031 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:16:53.556478 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.552820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:33063 (local address 127.0.0.1:37422)
1023 20:16:53.553479 (+ 659us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.553492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:53.553564 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:53.553888 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:53.553889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:53.553891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:53.554141 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.554144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.555208 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.555209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:53.556136 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.556139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.556221 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.556283 (+ 62us) client_negotiation.cc:769] Sending connection context
1023 20:16:53.556338 (+ 55us) client_negotiation.cc:240] Negotiation successful
1023 20:16:53.556384 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":70,"threads_started":1}
I20251023 20:16:53.556846 28491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.552814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:38905 (local address 127.0.0.1:56170)
1023 20:16:53.553201 (+ 387us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.553235 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:53.553324 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:53.553850 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:53.553853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:53.553857 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:53.554117 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.554120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.555750 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.555751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:53.556588 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.556592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.556664 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.556669 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:16:53.556702 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:16:53.556773 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":76,"threads_started":1}
I20251023 20:16:53.557156 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.553384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.9.117.193:33063)
1023 20:16:53.553698 (+ 314us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.553712 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.553724 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:16:53.553771 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.553774 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.553791 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.554219 (+ 428us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:53.554327 (+ 108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.555114 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.556226 (+ 1112us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.556877 (+ 651us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.556977 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.557011 (+ 34us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.557053 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20251023 20:16:53.557818 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.553476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.9.117.194:38905)
1023 20:16:53.553641 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.553648 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.553657 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:16:53.553702 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.553704 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.553723 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.554187 (+ 464us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:53.554272 (+ 85us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.555613 (+ 1341us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.557084 (+ 1471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.557583 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.557672 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.557705 (+ 33us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.557746 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20251023 20:16:53.558012 28154 tablet_service.cc:1505] Processing CreateTablet for tablet d3425e9163a04718bea1fde998896ce7 (DEFAULT_TABLE table=test-table [id=3eebf05f71a745aeb55c14acfa3781d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:53.558173 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.553282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.9.117.195:45879)
1023 20:16:53.553516 (+ 234us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.553525 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.553850 (+ 325us) server_negotiation.cc:408] Connection header received
1023 20:16:53.553897 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.553900 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.553924 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.553987 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:53.554344 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.556043 (+ 1699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.557290 (+ 1247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.557886 (+ 596us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.557975 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.558034 (+ 59us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.558082 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":116,"threads_started":1}
I20251023 20:16:53.558359 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.553034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:45879 (local address 127.0.0.1:41124)
1023 20:16:53.553711 (+ 677us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.553722 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:53.553785 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:53.554079 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:53.554080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:53.554082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:53.554252 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.554254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.556130 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.556131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:53.557051 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.557054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.557128 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.557131 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:53.557164 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:16:53.557205 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":91,"threads_started":1}
I20251023 20:16:53.558418 28154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3425e9163a04718bea1fde998896ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:53.561532 28496 tablet_bootstrap.cc:492] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a: Bootstrap starting.
I20251023 20:16:53.561404 28285 tablet_service.cc:1505] Processing CreateTablet for tablet d3425e9163a04718bea1fde998896ce7 (DEFAULT_TABLE table=test-table [id=3eebf05f71a745aeb55c14acfa3781d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:53.561569 28416 tablet_service.cc:1505] Processing CreateTablet for tablet d3425e9163a04718bea1fde998896ce7 (DEFAULT_TABLE table=test-table [id=3eebf05f71a745aeb55c14acfa3781d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:53.561779 28285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3425e9163a04718bea1fde998896ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:53.561833 28416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3425e9163a04718bea1fde998896ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:53.562165 28496 tablet_bootstrap.cc:654] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:53.562413 28496 log.cc:826] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:53.563167 28496 tablet_bootstrap.cc:492] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a: No bootstrap required, opened a new log
I20251023 20:16:53.563218 28496 ts_tablet_manager.cc:1403] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:16:53.564356 28496 raft_consensus.cc:359] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.564509 28496 raft_consensus.cc:385] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:53.564545 28496 raft_consensus.cc:740] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b3c16cba764b9f8fe77b972db0970a, State: Initialized, Role: FOLLOWER
I20251023 20:16:53.564666 28496 consensus_queue.cc:260] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.564973 28496 ts_tablet_manager.cc:1434] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:53.565132 28220 heartbeater.cc:499] Master 127.9.117.254:38281 was elected leader, sending a full tablet report...
I20251023 20:16:53.565188 28498 tablet_bootstrap.cc:492] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Bootstrap starting.
I20251023 20:16:53.565372 28500 tablet_bootstrap.cc:492] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: Bootstrap starting.
I20251023 20:16:53.566074 28498 tablet_bootstrap.cc:654] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:53.566075 28500 tablet_bootstrap.cc:654] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:53.566416 28498 log.cc:826] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:53.566416 28500 log.cc:826] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:53.567333 28500 tablet_bootstrap.cc:492] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: No bootstrap required, opened a new log
I20251023 20:16:53.567329 28498 tablet_bootstrap.cc:492] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: No bootstrap required, opened a new log
I20251023 20:16:53.567411 28498 ts_tablet_manager.cc:1403] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:16:53.567418 28500 ts_tablet_manager.cc:1403] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:16:53.569392 28500 raft_consensus.cc:359] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.569396 28498 raft_consensus.cc:359] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.569597 28500 raft_consensus.cc:385] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:53.569597 28498 raft_consensus.cc:385] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:53.569641 28498 raft_consensus.cc:740] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ffb5d229b7417eb7728b9389146156, State: Initialized, Role: FOLLOWER
I20251023 20:16:53.569636 28500 raft_consensus.cc:740] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa249608290e4f84b04331e71bdb85f1, State: Initialized, Role: FOLLOWER
I20251023 20:16:53.569787 28498 consensus_queue.cc:260] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.569787 28500 consensus_queue.cc:260] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.570122 28498 ts_tablet_manager.cc:1434] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:16:53.570122 28500 ts_tablet_manager.cc:1434] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:16:53.570185 28482 heartbeater.cc:499] Master 127.9.117.254:38281 was elected leader, sending a full tablet report...
I20251023 20:16:53.570281 28351 heartbeater.cc:499] Master 127.9.117.254:38281 was elected leader, sending a full tablet report...
W20251023 20:16:53.638983 28352 tablet.cc:2378] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:16:53.651114 28499 raft_consensus.cc:493] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:53.651280 28499 raft_consensus.cc:515] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.651690 28499 leader_election.cc:290] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20ffb5d229b7417eb7728b9389146156 (127.9.117.194:38905), aa249608290e4f84b04331e71bdb85f1 (127.9.117.195:45879)
I20251023 20:16:53.654661 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.651877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:38905 (local address 127.9.117.193:57553)
1023 20:16:53.651949 (+ 72us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.651963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:53.652022 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:53.652228 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:53.652229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:53.652231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:53.652473 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.652475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.653734 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.653735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:53.654411 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.654414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.654477 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.654480 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:53.654533 (+ 53us) client_negotiation.cc:240] Negotiation successful
1023 20:16:53.654564 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20251023 20:16:53.654804 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.651923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:45879 (local address 127.9.117.193:47637)
1023 20:16:53.652112 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.652124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:53.652206 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:53.652389 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:53.652390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:53.652392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:53.653101 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.653106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.653943 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.653944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:53.654552 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:53.654554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.654627 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.654630 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:53.654663 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:16:53.654700 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":111,"threads_started":1}
I20251023 20:16:53.655139 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.651959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:57553 (local address 127.9.117.194:38905)
1023 20:16:53.652025 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.652027 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.652037 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:53.652126 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.652128 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.652151 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.652222 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:53.652571 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.653640 (+ 1069us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.654498 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.655015 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.655026 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.655051 (+ 25us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.655091 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:16:53.655282 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.652013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:47637 (local address 127.9.117.195:45879)
1023 20:16:53.652104 (+ 91us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.652106 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.652203 (+ 97us) server_negotiation.cc:408] Connection header received
1023 20:16:53.652283 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.652285 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.652302 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.652359 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:53.653232 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.653833 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.654648 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.655145 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.655153 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.655173 (+ 20us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.655204 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251023 20:16:53.655498 28305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3425e9163a04718bea1fde998896ce7" candidate_uuid: "c5b3c16cba764b9f8fe77b972db0970a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ffb5d229b7417eb7728b9389146156" is_pre_election: true
I20251023 20:16:53.655712 28305 raft_consensus.cc:2468] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b3c16cba764b9f8fe77b972db0970a in term 0.
I20251023 20:16:53.655810 28436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3425e9163a04718bea1fde998896ce7" candidate_uuid: "c5b3c16cba764b9f8fe77b972db0970a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa249608290e4f84b04331e71bdb85f1" is_pre_election: true
I20251023 20:16:53.655973 28436 raft_consensus.cc:2468] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b3c16cba764b9f8fe77b972db0970a in term 0.
I20251023 20:16:53.655997 28108 leader_election.cc:304] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ffb5d229b7417eb7728b9389146156, c5b3c16cba764b9f8fe77b972db0970a; no voters:
I20251023 20:16:53.656126 28499 raft_consensus.cc:2804] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:16:53.656219 28499 raft_consensus.cc:493] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:16:53.656255 28499 raft_consensus.cc:3060] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:53.657058 28499 raft_consensus.cc:515] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.657236 28499 leader_election.cc:290] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [CANDIDATE]: Term 1 election: Requested vote from peers 20ffb5d229b7417eb7728b9389146156 (127.9.117.194:38905), aa249608290e4f84b04331e71bdb85f1 (127.9.117.195:45879)
I20251023 20:16:53.657454 28305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3425e9163a04718bea1fde998896ce7" candidate_uuid: "c5b3c16cba764b9f8fe77b972db0970a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ffb5d229b7417eb7728b9389146156"
I20251023 20:16:53.657541 28305 raft_consensus.cc:3060] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:53.658365 28305 raft_consensus.cc:2468] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b3c16cba764b9f8fe77b972db0970a in term 1.
I20251023 20:16:53.658480 28436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3425e9163a04718bea1fde998896ce7" candidate_uuid: "c5b3c16cba764b9f8fe77b972db0970a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa249608290e4f84b04331e71bdb85f1"
I20251023 20:16:53.658572 28108 leader_election.cc:304] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ffb5d229b7417eb7728b9389146156, c5b3c16cba764b9f8fe77b972db0970a; no voters:
I20251023 20:16:53.658576 28436 raft_consensus.cc:3060] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:53.658694 28499 raft_consensus.cc:2804] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:53.658874 28499 raft_consensus.cc:697] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [term 1 LEADER]: Becoming Leader. State: Replica: c5b3c16cba764b9f8fe77b972db0970a, State: Running, Role: LEADER
I20251023 20:16:53.658998 28499 consensus_queue.cc:237] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } }
I20251023 20:16:53.659454 28436 raft_consensus.cc:2468] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b3c16cba764b9f8fe77b972db0970a in term 1.
I20251023 20:16:53.659848 28032 catalog_manager.cc:5649] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b3c16cba764b9f8fe77b972db0970a (127.9.117.193). New cstate: current_term: 1 leader_uuid: "c5b3c16cba764b9f8fe77b972db0970a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b3c16cba764b9f8fe77b972db0970a" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 } health_report { overall_health: UNKNOWN } } }
I20251023 20:16:53.674934 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.672493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.9.117.193:33063)
1023 20:16:53.672564 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:53.672567 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:53.672576 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:16:53.672626 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:53.672627 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:53.672651 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:53.672700 (+ 49us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:16:53.673004 (+ 304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.673698 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:53.674338 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:53.674505 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:53.674553 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:16:53.674660 (+ 107us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:16:53.674733 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:53.674810 (+ 77us) server_negotiation.cc:300] Negotiation successful
1023 20:16:53.674844 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:16:53.676967 28305 raft_consensus.cc:1275] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 1 FOLLOWER]: Refusing update from remote peer c5b3c16cba764b9f8fe77b972db0970a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:53.676967 28436 raft_consensus.cc:1275] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 1 FOLLOWER]: Refusing update from remote peer c5b3c16cba764b9f8fe77b972db0970a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:53.677332 28506 consensus_queue.cc:1048] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa249608290e4f84b04331e71bdb85f1" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:53.677490 28499 consensus_queue.cc:1048] T d3425e9163a04718bea1fde998896ce7 P c5b3c16cba764b9f8fe77b972db0970a [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ffb5d229b7417eb7728b9389146156" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 38905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:53.680781 28511 mvcc.cc:204] Tried to move back new op lower bound from 7214082513617477632 to 7214082513547874304. Current Snapshot: MvccSnapshot[applied={T|T < 7214082513617477632}]
I20251023 20:16:53.682536 28032 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59528:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:53.682628 28032 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59528:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:53.683439 28032 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7eed2b3e25284b6fa6c5c078f7902efe: Sending DeleteTablet for 3 replicas of tablet d3425e9163a04718bea1fde998896ce7
I20251023 20:16:53.683802 28285 tablet_service.cc:1552] Processing DeleteTablet for tablet d3425e9163a04718bea1fde998896ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:53 UTC) from {username='slave'} at 127.0.0.1:56170
I20251023 20:16:53.683897 28154 tablet_service.cc:1552] Processing DeleteTablet for tablet d3425e9163a04718bea1fde998896ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:53 UTC) from {username='slave'} at 127.0.0.1:37422
I20251023 20:16:53.684047 28518 tablet_replica.cc:333] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: stopping tablet replica
I20251023 20:16:53.684060 28416 tablet_service.cc:1552] Processing DeleteTablet for tablet d3425e9163a04718bea1fde998896ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:53 UTC) from {username='slave'} at 127.0.0.1:41124
I20251023 20:16:53.684134 28518 raft_consensus.cc:2243] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:53.684243 28518 raft_consensus.cc:2272] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:53.684305 28520 tablet_replica.cc:333] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: stopping tablet replica
I20251023 20:16:53.684391 28520 raft_consensus.cc:2243] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:53.684401 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 28092
I20251023 20:16:53.684484 28520 raft_consensus.cc:2272] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:53.685081 28520 ts_tablet_manager.cc:1916] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:53.685388 28518 ts_tablet_manager.cc:1916] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:53.686720 28520 ts_tablet_manager.cc:1929] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:53.686830 28520 log.cc:1199] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/d3425e9163a04718bea1fde998896ce7
I20251023 20:16:53.687126 28520 ts_tablet_manager.cc:1950] T d3425e9163a04718bea1fde998896ce7 P aa249608290e4f84b04331e71bdb85f1: Deleting consensus metadata
I20251023 20:16:53.687505 28518 ts_tablet_manager.cc:1929] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:53.687656 28518 log.cc:1199] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/d3425e9163a04718bea1fde998896ce7
I20251023 20:16:53.687635 28018 catalog_manager.cc:4985] TS aa249608290e4f84b04331e71bdb85f1 (127.9.117.195:45879): tablet d3425e9163a04718bea1fde998896ce7 (table test-table [id=3eebf05f71a745aeb55c14acfa3781d3]) successfully deleted
I20251023 20:16:53.687940 28518 ts_tablet_manager.cc:1950] T d3425e9163a04718bea1fde998896ce7 P 20ffb5d229b7417eb7728b9389146156: Deleting consensus metadata
I20251023 20:16:53.688318 28017 catalog_manager.cc:4985] TS 20ffb5d229b7417eb7728b9389146156 (127.9.117.194:38905): tablet d3425e9163a04718bea1fde998896ce7 (table test-table [id=3eebf05f71a745aeb55c14acfa3781d3]) successfully deleted
W20251023 20:16:53.691103 28018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:33063 (error 108)
I20251023 20:16:53.691299 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 28223
I20251023 20:16:53.692243 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:53.692082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:33063 (local address 127.0.0.1:37442)
1023 20:16:53.692141 (+ 59us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:53.692195 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251023 20:16:53.692377 28018 catalog_manager.cc:4712] TS c5b3c16cba764b9f8fe77b972db0970a (127.9.117.193:33063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3425e9163a04718bea1fde998896ce7: Network error: Client connection negotiation failed: client connection to 127.9.117.193:33063: connect: Connection refused (error 111)
I20251023 20:16:53.696834 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 28354
I20251023 20:16:53.701463 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 28001
2025-10-23T20:16:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (803 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:16:53 dist-test-slave-x62p krb5kdc[28524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:16:53 dist-test-slave-x62p krb5kdc[28524](info): set up 2 sockets
Oct 23 20:16:53 dist-test-slave-x62p krb5kdc[28524](info): commencing operation
krb5kdc: starting...
W20251023 20:16:55.677040 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:16:55 dist-test-slave-x62p krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:16:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:16:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:55.821167 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:42183
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:55.905550 28540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:55.905778 28540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:55.905818 28540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:55.907368 28540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:16:55.907431 28540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:55.907454 28540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:55.907477 28540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:16:55.907505 28540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:16:55.909384 28540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:55.909680 28540 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:55.909881 28540 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:55.912559 28546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:55.912542 28548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:55.912518 28545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:55.913028 28540 server_base.cc:1047] running on GCE node
I20251023 20:16:55.913237 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:55.913533 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:55.914690 28540 hybrid_clock.cc:648] HybridClock initialized: now 1761250615914668 us; error 33 us; skew 500 ppm
Oct 23 20:16:55 dist-test-slave-x62p krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250615, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:55.917333 28540 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:16:55.917977 28540 webserver.cc:492] Webserver started at http://127.9.117.254:43019/ using document root <none> and password file <none>
I20251023 20:16:55.918181 28540 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:55.918222 28540 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:55.918340 28540 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:55.919370 28540 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "3a1d86caf6cc4d6595febd09277b8c2a"
format_stamp: "Formatted at 2025-10-23 20:16:55 on dist-test-slave-x62p"
server_key: "62a78c762786844dab668204c2dad4c7"
server_key_iv: "5f52f83a31f97c5a40447fd45c26cec5"
server_key_version: "encryptionkey@0"
I20251023 20:16:55.919695 28540 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "3a1d86caf6cc4d6595febd09277b8c2a"
format_stamp: "Formatted at 2025-10-23 20:16:55 on dist-test-slave-x62p"
server_key: "62a78c762786844dab668204c2dad4c7"
server_key_iv: "5f52f83a31f97c5a40447fd45c26cec5"
server_key_version: "encryptionkey@0"
I20251023 20:16:55.920972 28540 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251023 20:16:55.921720 28555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:55.921922 28540 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:55.922001 28540 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "3a1d86caf6cc4d6595febd09277b8c2a"
format_stamp: "Formatted at 2025-10-23 20:16:55 on dist-test-slave-x62p"
server_key: "62a78c762786844dab668204c2dad4c7"
server_key_iv: "5f52f83a31f97c5a40447fd45c26cec5"
server_key_version: "encryptionkey@0"
I20251023 20:16:55.922062 28540 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:55.930142 28540 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:55.935334 28540 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:55.935490 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:55.939771 28540 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:42183
I20251023 20:16:55.939829 28607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:42183 every 8 connection(s)
I20251023 20:16:55.940136 28540 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:16:55.940799 28608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:55.943369 28608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a: Bootstrap starting.
I20251023 20:16:55.943974 28608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:55.944218 28608 log.cc:826] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:55.944941 28608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a: No bootstrap required, opened a new log
I20251023 20:16:55.946174 28608 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1d86caf6cc4d6595febd09277b8c2a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42183 } }
I20251023 20:16:55.946303 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 28540
I20251023 20:16:55.946343 28608 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:55.946379 28608 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1d86caf6cc4d6595febd09277b8c2a, State: Initialized, Role: FOLLOWER
I20251023 20:16:55.946399 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:16:55.946516 28608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1d86caf6cc4d6595febd09277b8c2a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42183 } }
I20251023 20:16:55.946591 28608 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:16:55.946593 9687 external_mini_cluster.cc:1468] Setting key 488da65c0dacae67814ca82ee8f0feed
I20251023 20:16:55.946635 28608 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:16:55.946686 28608 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:55.947449 28608 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1d86caf6cc4d6595febd09277b8c2a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42183 } }
I20251023 20:16:55.947566 28608 leader_election.cc:304] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a1d86caf6cc4d6595febd09277b8c2a; no voters:
I20251023 20:16:55.947721 28608 leader_election.cc:290] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:16:55.947778 28613 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:55.947973 28613 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [term 1 LEADER]: Becoming Leader. State: Replica: 3a1d86caf6cc4d6595febd09277b8c2a, State: Running, Role: LEADER
I20251023 20:16:55.948007 28608 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:16:55.948146 28613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1d86caf6cc4d6595febd09277b8c2a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42183 } }
I20251023 20:16:55.948697 28614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a1d86caf6cc4d6595febd09277b8c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1d86caf6cc4d6595febd09277b8c2a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42183 } } }
I20251023 20:16:55.948709 28615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a1d86caf6cc4d6595febd09277b8c2a. Latest consensus state: current_term: 1 leader_uuid: "3a1d86caf6cc4d6595febd09277b8c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1d86caf6cc4d6595febd09277b8c2a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42183 } } }
I20251023 20:16:55.948810 28614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:55.948812 28615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:55.949019 28617 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:16:55.949815 28617 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:16:55.951390 28617 catalog_manager.cc:1357] Generated new cluster ID: 1964967def4a43db879c67e7a7722f8c
I20251023 20:16:55.951442 28617 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:16:55 dist-test-slave-x62p krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:55.956425 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:55.947060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.9.117.254:42183)
1023 20:16:55.947276 (+ 216us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:55.947288 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:55.947316 (+ 28us) server_negotiation.cc:408] Connection header received
1023 20:16:55.947432 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:55.947438 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:55.947491 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:55.947650 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:55.948699 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:55.950017 (+ 1318us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:55.950806 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:55.951079 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:55.953243 (+ 2164us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:55.953270 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:55.953270 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:55.953294 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:55.955086 (+ 1792us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:55.955405 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:55.955408 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:55.955409 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:55.955457 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:55.955612 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:55.955613 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:55.955613 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:55.955833 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:55.955972 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:55.956090 (+ 118us) server_negotiation.cc:300] Negotiation successful
1023 20:16:55.956233 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20251023 20:16:55.960462 28617 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:16:55.961369 28617 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:16:55.979218 28617 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a: Generated new TSK 0
I20251023 20:16:55.979466 28617 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:56.035753 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:42183
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:56.141569 28636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:56.141750 28636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:56.141779 28636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:56.143307 28636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:56.143357 28636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:56.143386 28636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:16:56.145169 28636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:42183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:56.145488 28636 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:56.145690 28636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:56.148561 28641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:56.148603 28644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:56.148773 28642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:56.149230 28636 server_base.cc:1047] running on GCE node
I20251023 20:16:56.149431 28636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:56.149664 28636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:56.150807 28636 hybrid_clock.cc:648] HybridClock initialized: now 1761250616150773 us; error 43 us; skew 500 ppm
Oct 23 20:16:56 dist-test-slave-x62p krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:56.153810 28636 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:16:56.154422 28636 webserver.cc:492] Webserver started at http://127.9.117.193:44805/ using document root <none> and password file <none>
I20251023 20:16:56.154624 28636 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:56.154675 28636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:56.154796 28636 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:56.155857 28636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "dfddbf41f95f42afbb6871e1fdb36dc0"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "b1b5062f3e49fc04d71fe3898da72d21"
server_key_iv: "e57e4f70e895ef9b17e8cf558a48f242"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.156193 28636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "dfddbf41f95f42afbb6871e1fdb36dc0"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "b1b5062f3e49fc04d71fe3898da72d21"
server_key_iv: "e57e4f70e895ef9b17e8cf558a48f242"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.157596 28636 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:16:56.158430 28651 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.158629 28636 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:16:56.158712 28636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "dfddbf41f95f42afbb6871e1fdb36dc0"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "b1b5062f3e49fc04d71fe3898da72d21"
server_key_iv: "e57e4f70e895ef9b17e8cf558a48f242"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.158800 28636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:56.188953 28636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:56.191134 28636 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:56.191288 28636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:56.191537 28636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:56.191869 28636 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:56.191907 28636 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.191942 28636 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:56.191970 28636 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.198407 28636 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:34475
I20251023 20:16:56.198499 28764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:34475 every 8 connection(s)
I20251023 20:16:56.198915 28636 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:16:56.202172 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 28636
I20251023 20:16:56.202309 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:16:56.202508 9687 external_mini_cluster.cc:1468] Setting key 9b9f2c051463d62efd35c9a3a78d070b
Oct 23 20:16:56 dist-test-slave-x62p krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:56.209726 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.199783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:58599 (local address 127.9.117.254:42183)
1023 20:16:56.199889 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.199893 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.200057 (+ 164us) server_negotiation.cc:408] Connection header received
1023 20:16:56.200643 (+ 586us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.200645 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.200666 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.200755 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:56.201638 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.202189 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.203441 (+ 1252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.203632 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.206117 (+ 2485us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:56.206123 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:56.206123 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:56.206134 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:56.208139 (+ 2005us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:56.208607 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:56.208609 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:56.208610 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:56.208645 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:56.208948 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:56.208949 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:56.208950 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:56.209097 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:56.209178 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.209472 (+ 294us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.209612 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251023 20:16:56.209774 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.199711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:42183 (local address 127.9.117.193:58599)
1023 20:16:56.199927 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.199963 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:56.200541 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:56.200808 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:56.200813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:56.201175 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:56.201528 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.201533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.202279 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.202281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:56.203364 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.203367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.203538 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.204337 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:56.204360 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:56.206011 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:56.208260 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:56.208268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:56.208271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:56.208520 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:56.208780 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:56.208781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:56.208782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:56.208871 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:56.209170 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:56.209175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:56.209321 (+ 146us) client_negotiation.cc:769] Sending connection context
1023 20:16:56.209419 (+ 98us) client_negotiation.cc:240] Negotiation successful
1023 20:16:56.209569 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
I20251023 20:16:56.210630 28765 heartbeater.cc:344] Connected to a master server at 127.9.117.254:42183
I20251023 20:16:56.210779 28765 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:56.211084 28765 heartbeater.cc:507] Master 127.9.117.254:42183 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:16:56.211728 28572 ts_manager.cc:194] Registered new tserver with Master: dfddbf41f95f42afbb6871e1fdb36dc0 (127.9.117.193:34475)
I20251023 20:16:56.212432 28572 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:58599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:56.253875 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:42183
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:56.336958 28772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:56.337150 28772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:56.337190 28772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:56.338665 28772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:56.338716 28772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:56.338759 28772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:16:56.340350 28772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:42183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:56.340631 28772 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:56.340865 28772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:56.343576 28780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:56.343585 28778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:56.343660 28777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:56.343741 28772 server_base.cc:1047] running on GCE node
I20251023 20:16:56.344036 28772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:56.344261 28772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:56.345405 28772 hybrid_clock.cc:648] HybridClock initialized: now 1761250616345387 us; error 29 us; skew 500 ppm
Oct 23 20:16:56 dist-test-slave-x62p krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:56.347954 28772 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:16:56.348574 28772 webserver.cc:492] Webserver started at http://127.9.117.194:46629/ using document root <none> and password file <none>
I20251023 20:16:56.348806 28772 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:56.348852 28772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:56.348990 28772 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:56.350093 28772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "ce6792ddf2ab486f8c1e2ab10d323f31"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "e1491c19af07ec8b8194cc1abd995d8c"
server_key_iv: "b7f3605f65295ec2ded95bc8625743a0"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.350415 28772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "ce6792ddf2ab486f8c1e2ab10d323f31"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "e1491c19af07ec8b8194cc1abd995d8c"
server_key_iv: "b7f3605f65295ec2ded95bc8625743a0"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.351771 28772 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:16:56.352597 28787 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.352869 28772 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:56.352954 28772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "ce6792ddf2ab486f8c1e2ab10d323f31"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "e1491c19af07ec8b8194cc1abd995d8c"
server_key_iv: "b7f3605f65295ec2ded95bc8625743a0"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.353026 28772 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:56.361804 28772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:56.363838 28772 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:56.363955 28772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:56.364185 28772 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:56.364498 28772 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:56.364530 28772 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.364565 28772 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:56.364591 28772 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.370797 28772 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:44911
I20251023 20:16:56.370880 28900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:44911 every 8 connection(s)
I20251023 20:16:56.371155 28772 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:16:56 dist-test-slave-x62p krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:56.378795 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 28772
I20251023 20:16:56.378954 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:16:56.379195 9687 external_mini_cluster.cc:1468] Setting key cb633633852dc6a1abbee63097b377a6
I20251023 20:16:56.381837 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.371989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:47651 (local address 127.9.117.254:42183)
1023 20:16:56.372057 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.372060 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.372243 (+ 183us) server_negotiation.cc:408] Connection header received
1023 20:16:56.372862 (+ 619us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.372864 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.372876 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.372939 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:56.373891 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.374425 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.375637 (+ 1212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.375821 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.378282 (+ 2461us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:56.378289 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:56.378289 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:56.378299 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:56.380153 (+ 1854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:56.380658 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:56.380660 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:56.380660 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:56.380696 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:56.381033 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:56.381034 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:56.381034 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:56.381190 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:56.381267 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.381624 (+ 357us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.381760 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251023 20:16:56.382123 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.371931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:42183 (local address 127.9.117.194:47651)
1023 20:16:56.372131 (+ 200us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.372162 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:56.372720 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:56.373049 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:56.373054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:56.373419 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:56.373773 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.373778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.374539 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.374540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:56.375519 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.375523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.375665 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.376284 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:56.376310 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:56.378154 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:56.380270 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:56.380280 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:56.380284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:56.380571 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:56.380865 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:56.380866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:56.380866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:56.380963 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:56.381279 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:56.381286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:56.381444 (+ 158us) client_negotiation.cc:769] Sending connection context
1023 20:16:56.381585 (+ 141us) client_negotiation.cc:240] Negotiation successful
1023 20:16:56.381844 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20251023 20:16:56.382673 28901 heartbeater.cc:344] Connected to a master server at 127.9.117.254:42183
I20251023 20:16:56.382773 28901 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:56.383016 28901 heartbeater.cc:507] Master 127.9.117.254:42183 requested a full tablet report, sending...
I20251023 20:16:56.383437 28572 ts_manager.cc:194] Registered new tserver with Master: ce6792ddf2ab486f8c1e2ab10d323f31 (127.9.117.194:44911)
I20251023 20:16:56.383862 28572 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:47651
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:56.427983 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:42183
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:56.511015 28908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:56.511207 28908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:56.511233 28908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:56.512857 28908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:56.512912 28908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:56.512941 28908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:16:56.514612 28908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:42183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:56.514860 28908 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:56.515061 28908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:56.517828 28916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:56.517819 28913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:56.517819 28914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:56.517887 28908 server_base.cc:1047] running on GCE node
I20251023 20:16:56.518293 28908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:56.518508 28908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:56.519650 28908 hybrid_clock.cc:648] HybridClock initialized: now 1761250616519638 us; error 29 us; skew 500 ppm
Oct 23 20:16:56 dist-test-slave-x62p krb5kdc[28524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:56.522226 28908 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:16:56.522787 28908 webserver.cc:492] Webserver started at http://127.9.117.195:43563/ using document root <none> and password file <none>
I20251023 20:16:56.522995 28908 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:56.523039 28908 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:56.523162 28908 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:56.524191 28908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "6a06dcfcf8c04a0f955b25e663e4c9af"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "08eabfcd4ea330fba6eb77b73f9cd0cd"
server_key_iv: "abc23e9757ecdc12bc1a5a78407db28e"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.524523 28908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "6a06dcfcf8c04a0f955b25e663e4c9af"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "08eabfcd4ea330fba6eb77b73f9cd0cd"
server_key_iv: "abc23e9757ecdc12bc1a5a78407db28e"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.525827 28908 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20251023 20:16:56.526652 28923 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.526876 28908 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:56.526953 28908 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "6a06dcfcf8c04a0f955b25e663e4c9af"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "08eabfcd4ea330fba6eb77b73f9cd0cd"
server_key_iv: "abc23e9757ecdc12bc1a5a78407db28e"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.527020 28908 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:56.537541 28908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:56.539528 28908 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:56.539677 28908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:56.539937 28908 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:56.540314 28908 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:56.540366 28908 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.540402 28908 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:56.540433 28908 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.547178 28908 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:34757
I20251023 20:16:56.547220 29036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:34757 every 8 connection(s)
I20251023 20:16:56.547559 28908 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:16:56.552886 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 28908
I20251023 20:16:56.553000 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:16:56.553227 9687 external_mini_cluster.cc:1468] Setting key 22c095e764891ad18cc15d9d15b6fae7
Oct 23 20:16:56 dist-test-slave-x62p krb5kdc[28524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:56.557303 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.548287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:48655 (local address 127.9.117.254:42183)
1023 20:16:56.548352 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.548355 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.548513 (+ 158us) server_negotiation.cc:408] Connection header received
1023 20:16:56.549103 (+ 590us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.549105 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.549119 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.549182 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:56.550041 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.550630 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.551697 (+ 1067us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.551876 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.554432 (+ 2556us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:56.554435 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:56.554435 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:56.554443 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:56.555995 (+ 1552us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:56.556393 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:56.556394 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:56.556395 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:56.556418 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:56.556615 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:56.556615 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:56.556616 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:56.556705 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:56.556831 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.557153 (+ 322us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.557236 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251023 20:16:56.557401 29039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.548232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:42183 (local address 127.9.117.195:48655)
1023 20:16:56.548406 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.548437 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:56.549004 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:56.549248 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:56.549252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:56.549638 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:56.549945 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.549948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.550720 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.550721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:56.551611 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.551613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.551758 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.552424 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:16:56.552446 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:56.554360 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:56.556088 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:56.556095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:56.556098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:56.556324 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:56.556478 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:16:56.556479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:16:56.556479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:16:56.556566 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:16:56.556835 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:56.556839 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:56.556979 (+ 140us) client_negotiation.cc:769] Sending connection context
1023 20:16:56.557097 (+ 118us) client_negotiation.cc:240] Negotiation successful
1023 20:16:56.557218 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20251023 20:16:56.557983 29037 heartbeater.cc:344] Connected to a master server at 127.9.117.254:42183
I20251023 20:16:56.558096 29037 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:56.558324 29037 heartbeater.cc:507] Master 127.9.117.254:42183 requested a full tablet report, sending...
I20251023 20:16:56.558749 28572 ts_manager.cc:194] Registered new tserver with Master: 6a06dcfcf8c04a0f955b25e663e4c9af (127.9.117.195:34757)
I20251023 20:16:56.559134 28572 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:48655
I20251023 20:16:56.565409 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:16:56.572360 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.566841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.9.117.254:42183)
1023 20:16:56.566903 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.566905 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.566916 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:16:56.566984 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.566984 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.566993 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.567047 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:56.567739 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.568222 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.568791 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.568911 (+ 120us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.570044 (+ 1133us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:56.570047 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:56.570048 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:56.570055 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:56.571419 (+ 1364us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:56.571788 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:56.571789 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:56.571789 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:56.571811 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:56.571938 (+ 127us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:56.571938 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:56.571938 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:56.572038 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:56.572118 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.572204 (+ 86us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.572253 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251023 20:16:56.573976 28572 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:16:56.575336 28572 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:16:56.580790 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.577714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:44911 (local address 127.0.0.1:60510)
1023 20:16:56.578096 (+ 382us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.578128 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:56.578221 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:56.578430 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:56.578432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:56.578436 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:56.578714 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.578716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.579801 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.579802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:56.580432 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.580434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.580515 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.580564 (+ 49us) client_negotiation.cc:769] Sending connection context
1023 20:16:56.580616 (+ 52us) client_negotiation.cc:240] Negotiation successful
1023 20:16:56.580673 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":110,"threads_started":1}
I20251023 20:16:56.580978 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.577714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34475 (local address 127.0.0.1:43428)
1023 20:16:56.578117 (+ 403us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.578131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:56.578222 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:56.578562 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:56.578563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:56.578565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:56.578766 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.578768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.580093 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.580094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:56.580703 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.580705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.580813 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.580819 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:16:56.580855 (+ 36us) client_negotiation.cc:240] Negotiation successful
1023 20:16:56.580897 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20251023 20:16:56.581538 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.577853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.9.117.193:34475)
1023 20:16:56.578056 (+ 203us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.578066 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.578207 (+ 141us) server_negotiation.cc:408] Connection header received
1023 20:16:56.578320 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.578323 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.578344 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.578414 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:56.578844 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.580014 (+ 1170us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.580844 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.581366 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.581411 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.581438 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.581468 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":106,"threads_started":1}
I20251023 20:16:56.581542 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.577853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.9.117.194:44911)
1023 20:16:56.578056 (+ 203us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.578066 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.578290 (+ 224us) server_negotiation.cc:408] Connection header received
1023 20:16:56.578334 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.578336 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.578350 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.578409 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:56.578820 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.579719 (+ 899us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.580536 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.581390 (+ 854us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.581427 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.581452 (+ 25us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.581485 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
I20251023 20:16:56.582093 28835 tablet_service.cc:1505] Processing CreateTablet for tablet 690487a922b1465abe3fc3bb97e75e49 (DEFAULT_TABLE table=test-table [id=7ef4f1a5c2804d5bbceb683d2e1687bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:56.582110 28699 tablet_service.cc:1505] Processing CreateTablet for tablet 690487a922b1465abe3fc3bb97e75e49 (DEFAULT_TABLE table=test-table [id=7ef4f1a5c2804d5bbceb683d2e1687bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:56.582441 28835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 690487a922b1465abe3fc3bb97e75e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:56.582444 28699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 690487a922b1465abe3fc3bb97e75e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:56.583200 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.577931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34757 (local address 127.0.0.1:35922)
1023 20:16:56.578333 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.578342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:56.578387 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:56.579058 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:56.579058 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:56.579060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:56.579246 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.579248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.582099 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.582101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:56.582973 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.582976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.583057 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.583062 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:16:56.583096 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:16:56.583132 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":93,"threads_started":1}
I20251023 20:16:56.583887 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.578067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.9.117.195:34757)
1023 20:16:56.578899 (+ 832us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.578907 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.578919 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:16:56.578960 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.578962 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.578978 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.579048 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:56.579331 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.582014 (+ 2683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.583098 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.583677 (+ 579us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.583728 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.583763 (+ 35us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.583813 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":111,"threads_started":1}
I20251023 20:16:56.584527 28971 tablet_service.cc:1505] Processing CreateTablet for tablet 690487a922b1465abe3fc3bb97e75e49 (DEFAULT_TABLE table=test-table [id=7ef4f1a5c2804d5bbceb683d2e1687bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:56.584836 28971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 690487a922b1465abe3fc3bb97e75e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:56.585332 29054 tablet_bootstrap.cc:492] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0: Bootstrap starting.
I20251023 20:16:56.585644 29055 tablet_bootstrap.cc:492] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Bootstrap starting.
I20251023 20:16:56.585991 29054 tablet_bootstrap.cc:654] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:56.586274 29054 log.cc:826] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:56.586295 29055 tablet_bootstrap.cc:654] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:56.586601 29055 log.cc:826] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:56.587014 29054 tablet_bootstrap.cc:492] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0: No bootstrap required, opened a new log
I20251023 20:16:56.587119 29054 ts_tablet_manager.cc:1403] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:56.587357 29055 tablet_bootstrap.cc:492] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: No bootstrap required, opened a new log
I20251023 20:16:56.587494 29055 ts_tablet_manager.cc:1403] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:56.587989 29058 tablet_bootstrap.cc:492] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: Bootstrap starting.
I20251023 20:16:56.588625 29058 tablet_bootstrap.cc:654] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:56.588969 29058 log.cc:826] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:56.589039 29055 raft_consensus.cc:359] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.589184 29055 raft_consensus.cc:385] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:56.589239 29055 raft_consensus.cc:740] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6792ddf2ab486f8c1e2ab10d323f31, State: Initialized, Role: FOLLOWER
I20251023 20:16:56.589186 29054 raft_consensus.cc:359] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.589380 29054 raft_consensus.cc:385] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:56.589365 29055 consensus_queue.cc:260] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.589470 29054 raft_consensus.cc:740] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfddbf41f95f42afbb6871e1fdb36dc0, State: Initialized, Role: FOLLOWER
I20251023 20:16:56.589624 29055 ts_tablet_manager.cc:1434] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:56.589597 29054 consensus_queue.cc:260] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.589735 29058 tablet_bootstrap.cc:492] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: No bootstrap required, opened a new log
I20251023 20:16:56.589655 28901 heartbeater.cc:499] Master 127.9.117.254:42183 was elected leader, sending a full tablet report...
I20251023 20:16:56.589797 29058 ts_tablet_manager.cc:1403] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:16:56.589854 29054 ts_tablet_manager.cc:1434] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:16:56.589987 28765 heartbeater.cc:499] Master 127.9.117.254:42183 was elected leader, sending a full tablet report...
I20251023 20:16:56.591300 29058 raft_consensus.cc:359] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.591398 29058 raft_consensus.cc:385] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:56.591415 29058 raft_consensus.cc:740] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a06dcfcf8c04a0f955b25e663e4c9af, State: Initialized, Role: FOLLOWER
I20251023 20:16:56.591490 29058 consensus_queue.cc:260] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.591727 29037 heartbeater.cc:499] Master 127.9.117.254:42183 was elected leader, sending a full tablet report...
I20251023 20:16:56.591800 29058 ts_tablet_manager.cc:1434] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20251023 20:16:56.621804 28902 tablet.cc:2378] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:16:56.698551 29061 raft_consensus.cc:493] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:56.698693 29061 raft_consensus.cc:515] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.699095 29061 leader_election.cc:290] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a06dcfcf8c04a0f955b25e663e4c9af (127.9.117.195:34757), ce6792ddf2ab486f8c1e2ab10d323f31 (127.9.117.194:44911)
W20251023 20:16:56.699661 28766 tablet.cc:2378] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:16:56.702253 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.699305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:34757 (local address 127.9.117.193:41393)
1023 20:16:56.699369 (+ 64us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.699383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:56.699457 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:56.699644 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:56.699645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:56.699646 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:56.699920 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.699922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.701000 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.701001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:56.702007 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.702009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.702073 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.702075 (+ 2us) client_negotiation.cc:769] Sending connection context
1023 20:16:56.702109 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:16:56.702160 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251023 20:16:56.702260 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.699318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:44911 (local address 127.9.117.193:34453)
1023 20:16:56.699476 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.699487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:56.699566 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:56.699699 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:56.699699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:56.699701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:56.699988 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.699990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.701000 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.701001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:56.701969 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:56.701973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.702068 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.702073 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:16:56.702109 (+ 36us) client_negotiation.cc:240] Negotiation successful
1023 20:16:56.702161 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20251023 20:16:56.702744 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.699345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:41393 (local address 127.9.117.195:34757)
1023 20:16:56.699391 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.699393 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.699488 (+ 95us) server_negotiation.cc:408] Connection header received
1023 20:16:56.699553 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.699554 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.699565 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.699621 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:56.700013 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.700890 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.702096 (+ 1206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.702605 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.702614 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.702641 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.702675 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:16:56.702744 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.699401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:34453 (local address 127.9.117.194:44911)
1023 20:16:56.699458 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.699460 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.699576 (+ 116us) server_negotiation.cc:408] Connection header received
1023 20:16:56.699620 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.699621 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.699632 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.699696 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:56.700077 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.700890 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.702087 (+ 1197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.702613 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.702621 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.702642 (+ 21us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.702676 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251023 20:16:56.703095 28855 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "690487a922b1465abe3fc3bb97e75e49" candidate_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" is_pre_election: true
I20251023 20:16:56.703095 28991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "690487a922b1465abe3fc3bb97e75e49" candidate_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" is_pre_election: true
I20251023 20:16:56.703249 28855 raft_consensus.cc:2468] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfddbf41f95f42afbb6871e1fdb36dc0 in term 0.
I20251023 20:16:56.703251 28991 raft_consensus.cc:2468] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfddbf41f95f42afbb6871e1fdb36dc0 in term 0.
I20251023 20:16:56.703502 28655 leader_election.cc:304] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a06dcfcf8c04a0f955b25e663e4c9af, dfddbf41f95f42afbb6871e1fdb36dc0; no voters:
I20251023 20:16:56.703678 29061 raft_consensus.cc:2804] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:16:56.703737 29061 raft_consensus.cc:493] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:16:56.703773 29061 raft_consensus.cc:3060] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:56.704512 29061 raft_consensus.cc:515] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.704712 29061 leader_election.cc:290] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [CANDIDATE]: Term 1 election: Requested vote from peers 6a06dcfcf8c04a0f955b25e663e4c9af (127.9.117.195:34757), ce6792ddf2ab486f8c1e2ab10d323f31 (127.9.117.194:44911)
I20251023 20:16:56.704931 28855 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "690487a922b1465abe3fc3bb97e75e49" candidate_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31"
I20251023 20:16:56.704928 28991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "690487a922b1465abe3fc3bb97e75e49" candidate_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af"
I20251023 20:16:56.705004 28991 raft_consensus.cc:3060] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:56.705010 28855 raft_consensus.cc:3060] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:56.705741 28855 raft_consensus.cc:2468] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfddbf41f95f42afbb6871e1fdb36dc0 in term 1.
I20251023 20:16:56.705739 28991 raft_consensus.cc:2468] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfddbf41f95f42afbb6871e1fdb36dc0 in term 1.
I20251023 20:16:56.705938 28654 leader_election.cc:304] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6a06dcfcf8c04a0f955b25e663e4c9af, ce6792ddf2ab486f8c1e2ab10d323f31, dfddbf41f95f42afbb6871e1fdb36dc0; no voters:
I20251023 20:16:56.706074 29061 raft_consensus.cc:2804] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:56.706254 29061 raft_consensus.cc:697] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [term 1 LEADER]: Becoming Leader. State: Replica: dfddbf41f95f42afbb6871e1fdb36dc0, State: Running, Role: LEADER
I20251023 20:16:56.706362 29061 consensus_queue.cc:237] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } }
I20251023 20:16:56.707191 28571 catalog_manager.cc:5649] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfddbf41f95f42afbb6871e1fdb36dc0 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfddbf41f95f42afbb6871e1fdb36dc0" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 34475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 } health_report { overall_health: UNKNOWN } } }
I20251023 20:16:56.729766 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.727349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.9.117.193:34475)
1023 20:16:56.727409 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.727411 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.727421 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:56.727535 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.727536 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.727556 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.727613 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:16:56.727900 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.728391 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.729055 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.729220 (+ 165us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.729392 (+ 172us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:16:56.729496 (+ 104us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:16:56.729583 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.729686 (+ 103us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.729697 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251023 20:16:56.731586 28991 raft_consensus.cc:1275] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 1 FOLLOWER]: Refusing update from remote peer dfddbf41f95f42afbb6871e1fdb36dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:56.731586 28855 raft_consensus.cc:1275] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 1 FOLLOWER]: Refusing update from remote peer dfddbf41f95f42afbb6871e1fdb36dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:56.731946 29064 consensus_queue.cc:1048] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a06dcfcf8c04a0f955b25e663e4c9af" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 34757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:56.732077 29061 consensus_queue.cc:1048] T 690487a922b1465abe3fc3bb97e75e49 P dfddbf41f95f42afbb6871e1fdb36dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6792ddf2ab486f8c1e2ab10d323f31" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:56.735020 29069 mvcc.cc:204] Tried to move back new op lower bound from 7214082526129815552 to 7214082526029950976. Current Snapshot: MvccSnapshot[applied={T|T < 7214082526129815552 or (T in {7214082526129815552})}]
I20251023 20:16:56.737434 28572 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51712:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:56.737541 28572 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51712:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:56.738330 28572 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3a1d86caf6cc4d6595febd09277b8c2a: Sending DeleteTablet for 3 replicas of tablet 690487a922b1465abe3fc3bb97e75e49
I20251023 20:16:56.738646 28699 tablet_service.cc:1552] Processing DeleteTablet for tablet 690487a922b1465abe3fc3bb97e75e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:43428
I20251023 20:16:56.738718 28971 tablet_service.cc:1552] Processing DeleteTablet for tablet 690487a922b1465abe3fc3bb97e75e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:35922
I20251023 20:16:56.738713 28835 tablet_service.cc:1552] Processing DeleteTablet for tablet 690487a922b1465abe3fc3bb97e75e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:56 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:60510
I20251023 20:16:56.739043 29079 tablet_replica.cc:333] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: stopping tablet replica
I20251023 20:16:56.739142 29079 raft_consensus.cc:2243] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:56.739235 29079 raft_consensus.cc:2272] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:56.739313 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 28636
I20251023 20:16:56.739907 29078 tablet_replica.cc:333] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: stopping tablet replica
I20251023 20:16:56.739979 29078 raft_consensus.cc:2243] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:56.740051 29078 raft_consensus.cc:2272] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:56.740417 29078 ts_tablet_manager.cc:1916] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:56.742022 29079 ts_tablet_manager.cc:1916] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:56.742399 29078 ts_tablet_manager.cc:1929] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:56.742506 29078 log.cc:1199] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/690487a922b1465abe3fc3bb97e75e49
I20251023 20:16:56.742801 29078 ts_tablet_manager.cc:1950] T 690487a922b1465abe3fc3bb97e75e49 P 6a06dcfcf8c04a0f955b25e663e4c9af: Deleting consensus metadata
I20251023 20:16:56.743273 28559 catalog_manager.cc:4985] TS 6a06dcfcf8c04a0f955b25e663e4c9af (127.9.117.195:34757): tablet 690487a922b1465abe3fc3bb97e75e49 (table test-table [id=7ef4f1a5c2804d5bbceb683d2e1687bd]) successfully deleted
I20251023 20:16:56.743490 29079 ts_tablet_manager.cc:1929] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:56.743563 29079 log.cc:1199] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/690487a922b1465abe3fc3bb97e75e49
I20251023 20:16:56.743798 29079 ts_tablet_manager.cc:1950] T 690487a922b1465abe3fc3bb97e75e49 P ce6792ddf2ab486f8c1e2ab10d323f31: Deleting consensus metadata
I20251023 20:16:56.744189 28558 catalog_manager.cc:4985] TS ce6792ddf2ab486f8c1e2ab10d323f31 (127.9.117.194:44911): tablet 690487a922b1465abe3fc3bb97e75e49 (table test-table [id=7ef4f1a5c2804d5bbceb683d2e1687bd]) successfully deleted
W20251023 20:16:56.746024 28559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:34475 (error 108)
I20251023 20:16:56.746260 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 28772
I20251023 20:16:56.747247 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.747100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:34475 (local address 127.0.0.1:43458)
1023 20:16:56.747177 (+ 77us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:56.747216 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20251023 20:16:56.747390 28559 catalog_manager.cc:4712] TS dfddbf41f95f42afbb6871e1fdb36dc0 (127.9.117.193:34475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 690487a922b1465abe3fc3bb97e75e49: Network error: Client connection negotiation failed: client connection to 127.9.117.193:34475: connect: Connection refused (error 111)
I20251023 20:16:56.752261 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 28908
I20251023 20:16:56.757545 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 28540
2025-10-23T20:16:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3056 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2025-10-23T20:16:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:16:56Z Disabled control of system clock
I20251023 20:16:56.785363 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:38397
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251023 20:16:56.867980 29084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:56.868170 29084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:56.868191 29084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:56.869714 29084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:16:56.869766 29084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:56.869779 29084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:56.869792 29084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:16:56.869805 29084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:16:56.871356 29084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:38397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:56.871600 29084 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:56.871816 29084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:56.874459 29092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:56.874678 29084 server_base.cc:1047] running on GCE node
W20251023 20:16:56.874446 29090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:56.874446 29089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:56.875006 29084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:56.875260 29084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:56.876425 29084 hybrid_clock.cc:648] HybridClock initialized: now 1761250616876408 us; error 40 us; skew 500 ppm
I20251023 20:16:56.877643 29084 webserver.cc:492] Webserver started at http://127.9.117.254:34537/ using document root <none> and password file <none>
I20251023 20:16:56.877856 29084 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:56.877905 29084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:56.878029 29084 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:56.879042 29084 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "288c6220575b40e39a0f2e29b2f6b839"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "27c534972782142fb46b2a1bec305e11"
server_key_iv: "420093052b9793f73097acb5455665b6"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.879395 29084 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "288c6220575b40e39a0f2e29b2f6b839"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "27c534972782142fb46b2a1bec305e11"
server_key_iv: "420093052b9793f73097acb5455665b6"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.880631 29084 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:16:56.881415 29098 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:56.881615 29084 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:16:56.881691 29084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "288c6220575b40e39a0f2e29b2f6b839"
format_stamp: "Formatted at 2025-10-23 20:16:56 on dist-test-slave-x62p"
server_key: "27c534972782142fb46b2a1bec305e11"
server_key_iv: "420093052b9793f73097acb5455665b6"
server_key_version: "encryptionkey@0"
I20251023 20:16:56.881755 29084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:56.900491 29084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:56.900889 29084 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:56.901027 29084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:56.905371 29084 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:38397
I20251023 20:16:56.905416 29150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:38397 every 8 connection(s)
I20251023 20:16:56.905789 29084 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:16:56.906411 29151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:56.909076 29151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839: Bootstrap starting.
I20251023 20:16:56.909689 29151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:56.909907 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 29084
I20251023 20:16:56.909950 29151 log.cc:826] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:56.910019 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:16:56.910199 9687 external_mini_cluster.cc:1468] Setting key 0def1ebd0da83e059e410031c61a743b
I20251023 20:16:56.910979 29151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839: No bootstrap required, opened a new log
I20251023 20:16:56.912508 29151 raft_consensus.cc:359] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288c6220575b40e39a0f2e29b2f6b839" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38397 } }
I20251023 20:16:56.912648 29151 raft_consensus.cc:385] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:56.912675 29151 raft_consensus.cc:740] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288c6220575b40e39a0f2e29b2f6b839, State: Initialized, Role: FOLLOWER
I20251023 20:16:56.912802 29151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288c6220575b40e39a0f2e29b2f6b839" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38397 } }
I20251023 20:16:56.912873 29151 raft_consensus.cc:399] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:16:56.912923 29151 raft_consensus.cc:493] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:16:56.912973 29151 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:56.913592 29151 raft_consensus.cc:515] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288c6220575b40e39a0f2e29b2f6b839" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38397 } }
I20251023 20:16:56.913702 29151 leader_election.cc:304] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 288c6220575b40e39a0f2e29b2f6b839; no voters:
I20251023 20:16:56.913923 29151 leader_election.cc:290] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:16:56.913996 29156 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:56.914181 29156 raft_consensus.cc:697] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [term 1 LEADER]: Becoming Leader. State: Replica: 288c6220575b40e39a0f2e29b2f6b839, State: Running, Role: LEADER
I20251023 20:16:56.914238 29151 sys_catalog.cc:565] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:16:56.914307 29156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288c6220575b40e39a0f2e29b2f6b839" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38397 } }
I20251023 20:16:56.914722 29157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "288c6220575b40e39a0f2e29b2f6b839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288c6220575b40e39a0f2e29b2f6b839" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38397 } } }
I20251023 20:16:56.914963 29157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:56.915166 29165 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:16:56.915340 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:56.910740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.9.117.254:38397)
1023 20:16:56.910954 (+ 214us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:56.910961 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:56.910979 (+ 18us) server_negotiation.cc:408] Connection header received
1023 20:16:56.911367 (+ 388us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:56.911373 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:56.911426 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:56.911549 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:56.912077 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.912986 (+ 909us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:56.913597 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:56.913888 (+ 291us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:56.914398 (+ 510us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:56.914407 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:56.914408 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:56.914638 (+ 230us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:56.914676 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:56.914687 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:56.915066 (+ 379us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:56.915109 (+ 43us) server_negotiation.cc:300] Negotiation successful
1023 20:16:56.915173 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20251023 20:16:56.914711 29156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 288c6220575b40e39a0f2e29b2f6b839. Latest consensus state: current_term: 1 leader_uuid: "288c6220575b40e39a0f2e29b2f6b839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288c6220575b40e39a0f2e29b2f6b839" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 38397 } } }
I20251023 20:16:56.916172 29156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839 [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:56.915675 29165 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:16:56.918126 29165 catalog_manager.cc:1357] Generated new cluster ID: 10c0c9af123f48f59c5e947e4475d330
I20251023 20:16:56.918185 29165 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:16:56.926401 29165 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:16:56.927026 29165 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:16:56.934369 29165 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839: Generated new TSK 0
I20251023 20:16:56.934595 29165 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:16:56.942163 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38397
--builtin_ntp_servers=127.9.117.212:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251023 20:16:57.028198 29175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:57.028391 29175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:57.028416 29175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:57.030218 29175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:57.030282 29175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:57.030313 29175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:16:57.031941 29175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:57.032224 29175 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:57.032482 29175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:57.035282 29180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:57.035290 29183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:57.035378 29181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:57.035564 29175 server_base.cc:1047] running on GCE node
I20251023 20:16:57.035763 29175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:57.036033 29175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:57.037204 29175 hybrid_clock.cc:648] HybridClock initialized: now 1761250617037178 us; error 42 us; skew 500 ppm
I20251023 20:16:57.038836 29175 webserver.cc:492] Webserver started at http://127.9.117.193:42811/ using document root <none> and password file <none>
I20251023 20:16:57.039103 29175 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:57.039160 29175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:57.039288 29175 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:57.040704 29175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "cf015b0894584703bfb4f916f1834684"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "11c073f64248eb54a24b7563c526b358"
server_key_iv: "20094c04b01ec666218624f4459c4ffa"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.041198 29175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "cf015b0894584703bfb4f916f1834684"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "11c073f64248eb54a24b7563c526b358"
server_key_iv: "20094c04b01ec666218624f4459c4ffa"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.042862 29175 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251023 20:16:57.043885 29189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.044112 29175 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:16:57.044201 29175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "cf015b0894584703bfb4f916f1834684"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "11c073f64248eb54a24b7563c526b358"
server_key_iv: "20094c04b01ec666218624f4459c4ffa"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.044279 29175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:57.064064 29175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:57.064481 29175 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:57.064638 29175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:57.064992 29175 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:57.065404 29175 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:57.065460 29175 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.065495 29175 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:57.065513 29175 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.074005 29175 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:41373
I20251023 20:16:57.074184 29302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:41373 every 8 connection(s)
I20251023 20:16:57.074504 29175 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:16:57.077294 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 29175
I20251023 20:16:57.077414 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:16:57.077625 9687 external_mini_cluster.cc:1468] Setting key 3bea59dc6862c17e88615f49ef0c9972
I20251023 20:16:57.078678 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38397
--builtin_ntp_servers=127.9.117.212:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251023 20:16:57.080351 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.075528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:57415 (local address 127.9.117.254:38397)
1023 20:16:57.075602 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.075605 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.075842 (+ 237us) server_negotiation.cc:408] Connection header received
1023 20:16:57.076358 (+ 516us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.076360 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.076375 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.076461 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:57.077032 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.077852 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.078756 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.078950 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.080026 (+ 1076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:57.080031 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:57.080032 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:57.080039 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:57.080057 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:57.080061 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:57.080089 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.080218 (+ 129us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.080263 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:16:57.080863 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.075486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:38397 (local address 127.9.117.193:57415)
1023 20:16:57.075735 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.075760 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:57.076269 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:57.076540 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:57.076544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:57.076554 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:57.076942 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.076946 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.077987 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.077989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:57.078646 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.078649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.079116 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.079857 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:16:57.079858 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:57.079896 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:16:57.079921 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:57.080130 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:57.080135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:57.080138 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:57.080527 (+ 389us) client_negotiation.cc:240] Negotiation successful
1023 20:16:57.080641 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20251023 20:16:57.081818 29303 heartbeater.cc:344] Connected to a master server at 127.9.117.254:38397
I20251023 20:16:57.081960 29303 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:57.082262 29303 heartbeater.cc:507] Master 127.9.117.254:38397 requested a full tablet report, sending...
I20251023 20:16:57.082983 29115 ts_manager.cc:194] Registered new tserver with Master: cf015b0894584703bfb4f916f1834684 (127.9.117.193:41373)
I20251023 20:16:57.084151 29115 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.193:57415
W20251023 20:16:57.167670 29306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:57.167876 29306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:57.167917 29306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:57.169584 29306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:57.169637 29306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:57.169683 29306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:16:57.171303 29306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:57.171626 29306 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:57.171895 29306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:57.174762 29314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:57.174801 29306 server_base.cc:1047] running on GCE node
W20251023 20:16:57.174958 29312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:57.174739 29311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:57.175177 29306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:57.175388 29306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:57.176563 29306 hybrid_clock.cc:648] HybridClock initialized: now 1761250617176545 us; error 34 us; skew 500 ppm
I20251023 20:16:57.177918 29306 webserver.cc:492] Webserver started at http://127.9.117.194:33531/ using document root <none> and password file <none>
I20251023 20:16:57.178141 29306 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:57.178193 29306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:57.178325 29306 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:57.179404 29306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "34ce696027a74825a07b0ed5f2f27e0f"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "1379e14371d964762122bfbdfa6c6927"
server_key_iv: "7b47c66ff51119d7f0b9eb253f8f0523"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.179740 29306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "34ce696027a74825a07b0ed5f2f27e0f"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "1379e14371d964762122bfbdfa6c6927"
server_key_iv: "7b47c66ff51119d7f0b9eb253f8f0523"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.181074 29306 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:16:57.181950 29320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.182188 29306 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:16:57.182276 29306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "34ce696027a74825a07b0ed5f2f27e0f"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "1379e14371d964762122bfbdfa6c6927"
server_key_iv: "7b47c66ff51119d7f0b9eb253f8f0523"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.182349 29306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:57.196033 29306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:57.196568 29306 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:57.196717 29306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:57.197105 29306 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:57.197503 29306 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:57.197542 29306 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.197578 29306 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:57.197598 29306 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.204167 29306 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:36887
I20251023 20:16:57.204234 29433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:36887 every 8 connection(s)
I20251023 20:16:57.204592 29306 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:16:57.209273 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.205423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:41387 (local address 127.9.117.254:38397)
1023 20:16:57.205511 (+ 88us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.205514 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.205723 (+ 209us) server_negotiation.cc:408] Connection header received
1023 20:16:57.206295 (+ 572us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.206297 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.206313 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.206367 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:57.206911 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.207419 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.208028 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.208200 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.208968 (+ 768us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:57.208974 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:57.208974 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:57.208983 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:57.209000 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:57.209003 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:57.209053 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.209146 (+ 93us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.209188 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251023 20:16:57.209450 29436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.205370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:38397 (local address 127.9.117.194:41387)
1023 20:16:57.205615 (+ 245us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.205642 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:57.206196 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:57.206446 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:57.206451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:57.206460 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:57.206816 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.206818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.207528 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.207529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:57.207947 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.207949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.208107 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.208831 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:16:57.208832 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:57.208863 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:16:57.208885 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:57.209060 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:57.209065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:57.209068 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:57.209138 (+ 70us) client_negotiation.cc:240] Negotiation successful
1023 20:16:57.209242 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20251023 20:16:57.210109 29434 heartbeater.cc:344] Connected to a master server at 127.9.117.254:38397
I20251023 20:16:57.210227 29434 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:57.210476 29434 heartbeater.cc:507] Master 127.9.117.254:38397 requested a full tablet report, sending...
I20251023 20:16:57.211001 29115 ts_manager.cc:194] Registered new tserver with Master: 34ce696027a74825a07b0ed5f2f27e0f (127.9.117.194:36887)
I20251023 20:16:57.211401 29115 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.194:41387
I20251023 20:16:57.214726 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 29306
I20251023 20:16:57.214846 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:16:57.214995 9687 external_mini_cluster.cc:1468] Setting key 3953cb695bf34e5c0b089597d046430d
I20251023 20:16:57.216212 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38397
--builtin_ntp_servers=127.9.117.212:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251023 20:16:57.303114 29437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:57.303323 29437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:57.303347 29437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:57.304976 29437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:57.305037 29437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:57.305068 29437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:16:57.306705 29437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:57.306959 29437 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:57.307185 29437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:16:57.310111 29443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:57.310176 29437 server_base.cc:1047] running on GCE node
W20251023 20:16:57.310088 29442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:57.310088 29445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:57.310520 29437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:57.310734 29437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:57.311894 29437 hybrid_clock.cc:648] HybridClock initialized: now 1761250617311874 us; error 40 us; skew 500 ppm
I20251023 20:16:57.313344 29437 webserver.cc:492] Webserver started at http://127.9.117.195:40487/ using document root <none> and password file <none>
I20251023 20:16:57.313580 29437 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:57.313634 29437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:57.313752 29437 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:57.314894 29437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "5534a1238db447bba47ff097a938c107"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "cc243a523e3e19eaf4b58f9e40ae1d1e"
server_key_iv: "33ec26adce45053c3a1a1f7d644d6c16"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.315220 29437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "5534a1238db447bba47ff097a938c107"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "cc243a523e3e19eaf4b58f9e40ae1d1e"
server_key_iv: "33ec26adce45053c3a1a1f7d644d6c16"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.316591 29437 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251023 20:16:57.317411 29451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.317585 29437 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:16:57.317659 29437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "5534a1238db447bba47ff097a938c107"
format_stamp: "Formatted at 2025-10-23 20:16:57 on dist-test-slave-x62p"
server_key: "cc243a523e3e19eaf4b58f9e40ae1d1e"
server_key_iv: "33ec26adce45053c3a1a1f7d644d6c16"
server_key_version: "encryptionkey@0"
I20251023 20:16:57.317739 29437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:57.333716 29437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:57.334089 29437 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:57.334221 29437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:57.334470 29437 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:16:57.334839 29437 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:16:57.334877 29437 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.334914 29437 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:16:57.334942 29437 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:57.341226 29437 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:45189
I20251023 20:16:57.341329 29564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:45189 every 8 connection(s)
I20251023 20:16:57.341651 29437 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:16:57.346091 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.342404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:32951 (local address 127.9.117.254:38397)
1023 20:16:57.342467 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.342470 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.342752 (+ 282us) server_negotiation.cc:408] Connection header received
1023 20:16:57.343255 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.343256 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.343269 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.343346 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:57.343871 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.344382 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.345084 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.345230 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.345833 (+ 603us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:57.345838 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:57.345838 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:57.345844 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:57.345863 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:57.345866 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:57.345916 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.345990 (+ 74us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.346022 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:16:57.346231 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.342358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:38397 (local address 127.9.117.195:32951)
1023 20:16:57.342604 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.342636 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:57.343164 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:57.343404 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:57.343408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:57.343417 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:16:57.343766 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.343769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.344490 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.344491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:57.345001 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.345004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.345130 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.345703 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:16:57.345703 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:16:57.345728 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:16:57.345748 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:16:57.345922 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:16:57.345925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:16:57.345928 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:57.345987 (+ 59us) client_negotiation.cc:240] Negotiation successful
1023 20:16:57.346062 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20251023 20:16:57.346830 29565 heartbeater.cc:344] Connected to a master server at 127.9.117.254:38397
I20251023 20:16:57.346957 29565 heartbeater.cc:461] Registering TS with master...
I20251023 20:16:57.347205 29565 heartbeater.cc:507] Master 127.9.117.254:38397 requested a full tablet report, sending...
I20251023 20:16:57.347641 29115 ts_manager.cc:194] Registered new tserver with Master: 5534a1238db447bba47ff097a938c107 (127.9.117.195:45189)
I20251023 20:16:57.348016 29115 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.195:32951
I20251023 20:16:57.351811 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 29437
I20251023 20:16:57.351933 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:16:57.352129 9687 external_mini_cluster.cc:1468] Setting key e60e1078141433c0de9fa5b46a843734
I20251023 20:16:57.353396 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:16:57.358076 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.354968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.9.117.254:38397)
1023 20:16:57.355044 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.355047 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.355141 (+ 94us) server_negotiation.cc:408] Connection header received
1023 20:16:57.355234 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.355236 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.355253 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.355325 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:57.355720 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.356243 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.356903 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.357107 (+ 204us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.357705 (+ 598us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:57.357710 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:57.357711 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:16:57.357718 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:57.357737 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:16:57.357742 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:57.357857 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.357964 (+ 107us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.357983 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251023 20:16:57.359947 29115 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:16:57.361495 29115 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:16:57.367412 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.363744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:45189 (local address 127.0.0.1:46182)
1023 20:16:57.363952 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.363964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:57.364033 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:57.364781 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:57.364784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:57.364787 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:57.365036 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.365039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.366239 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.366240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:57.367098 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.367100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.367174 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.367223 (+ 49us) client_negotiation.cc:769] Sending connection context
1023 20:16:57.367276 (+ 53us) client_negotiation.cc:240] Negotiation successful
1023 20:16:57.367319 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":55,"thread_start_us":74,"threads_started":1}
I20251023 20:16:57.367823 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.363743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:41373 (local address 127.0.0.1:47872)
1023 20:16:57.363912 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.363946 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:57.364033 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:57.364931 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:57.364932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:57.364933 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:57.365103 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.365104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.366773 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.366774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:57.367610 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.367613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.367673 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.367676 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:16:57.367705 (+ 29us) client_negotiation.cc:240] Negotiation successful
1023 20:16:57.367741 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20251023 20:16:57.367743 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.364004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:36887 (local address 127.0.0.1:35962)
1023 20:16:57.364240 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.364251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:57.364305 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:57.364784 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:57.364786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:57.364788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:57.365075 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.365077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.366714 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.366717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:57.367507 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.367510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.367594 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.367598 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:16:57.367632 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:16:57.367673 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":72,"threads_started":1}
I20251023 20:16:57.368021 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.363861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.9.117.195:45189)
1023 20:16:57.364436 (+ 575us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.364443 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.364453 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:57.364516 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.364519 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.364536 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.364608 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:57.365120 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.366146 (+ 1026us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.367186 (+ 1040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.367733 (+ 547us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.367862 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.367897 (+ 35us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.367947 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":123,"threads_started":1}
I20251023 20:16:57.368551 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.364486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.9.117.193:41373)
1023 20:16:57.364754 (+ 268us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.364763 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.364773 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:57.364811 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.364814 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.364829 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.365174 (+ 345us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:57.365286 (+ 112us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.366480 (+ 1194us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.367783 (+ 1303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.368359 (+ 576us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.368424 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.368450 (+ 26us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.368481 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":131,"threads_started":1}
I20251023 20:16:57.368750 29499 tablet_service.cc:1505] Processing CreateTablet for tablet f9735c8d89f2438a929ce68020f7a5d3 (DEFAULT_TABLE table=test-table [id=dcda1f8a0a494af080bb6e9d2967cc01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:57.368979 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.364248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.9.117.194:36887)
1023 20:16:57.364466 (+ 218us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.364476 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.364487 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:16:57.364536 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.364538 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.364565 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.364645 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:57.365165 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.366629 (+ 1464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.367965 (+ 1336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.368713 (+ 748us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.368821 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.368855 (+ 34us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.368898 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":96,"threads_started":1}
I20251023 20:16:57.369149 29499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9735c8d89f2438a929ce68020f7a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:57.369155 29237 tablet_service.cc:1505] Processing CreateTablet for tablet f9735c8d89f2438a929ce68020f7a5d3 (DEFAULT_TABLE table=test-table [id=dcda1f8a0a494af080bb6e9d2967cc01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:57.369535 29237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9735c8d89f2438a929ce68020f7a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:57.369648 29368 tablet_service.cc:1505] Processing CreateTablet for tablet f9735c8d89f2438a929ce68020f7a5d3 (DEFAULT_TABLE table=test-table [id=dcda1f8a0a494af080bb6e9d2967cc01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:16:57.369985 29368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9735c8d89f2438a929ce68020f7a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:57.372002 29579 tablet_bootstrap.cc:492] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684: Bootstrap starting.
I20251023 20:16:57.372759 29579 tablet_bootstrap.cc:654] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:57.373139 29579 log.cc:826] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:57.373083 29580 tablet_bootstrap.cc:492] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Bootstrap starting.
I20251023 20:16:57.373262 29581 tablet_bootstrap.cc:492] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Bootstrap starting.
I20251023 20:16:57.373857 29580 tablet_bootstrap.cc:654] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:57.374053 29579 tablet_bootstrap.cc:492] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684: No bootstrap required, opened a new log
I20251023 20:16:57.374121 29579 ts_tablet_manager.cc:1403] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:16:57.374138 29581 tablet_bootstrap.cc:654] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:57.374181 29580 log.cc:826] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:57.374436 29581 log.cc:826] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:57.375036 29580 tablet_bootstrap.cc:492] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: No bootstrap required, opened a new log
I20251023 20:16:57.375105 29580 ts_tablet_manager.cc:1403] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:57.375555 29579 raft_consensus.cc:359] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.375681 29579 raft_consensus.cc:385] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:57.375700 29579 raft_consensus.cc:740] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf015b0894584703bfb4f916f1834684, State: Initialized, Role: FOLLOWER
I20251023 20:16:57.375828 29579 consensus_queue.cc:260] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.376135 29579 ts_tablet_manager.cc:1434] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:57.376257 29303 heartbeater.cc:499] Master 127.9.117.254:38397 was elected leader, sending a full tablet report...
I20251023 20:16:57.376621 29580 raft_consensus.cc:359] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.376801 29580 raft_consensus.cc:385] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:57.376834 29580 raft_consensus.cc:740] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5534a1238db447bba47ff097a938c107, State: Initialized, Role: FOLLOWER
I20251023 20:16:57.376946 29580 consensus_queue.cc:260] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.377172 29581 tablet_bootstrap.cc:492] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: No bootstrap required, opened a new log
I20251023 20:16:57.377230 29580 ts_tablet_manager.cc:1434] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:57.377280 29581 ts_tablet_manager.cc:1403] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:16:57.377244 29565 heartbeater.cc:499] Master 127.9.117.254:38397 was elected leader, sending a full tablet report...
I20251023 20:16:57.378670 29581 raft_consensus.cc:359] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.378790 29581 raft_consensus.cc:385] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:57.378814 29581 raft_consensus.cc:740] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ce696027a74825a07b0ed5f2f27e0f, State: Initialized, Role: FOLLOWER
I20251023 20:16:57.378892 29581 consensus_queue.cc:260] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.379148 29581 ts_tablet_manager.cc:1434] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:16:57.379251 29434 heartbeater.cc:499] Master 127.9.117.254:38397 was elected leader, sending a full tablet report...
W20251023 20:16:57.455209 29435 tablet.cc:2378] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:16:57.543265 29585 raft_consensus.cc:493] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:16:57.543402 29585 raft_consensus.cc:515] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.543735 29585 leader_election.cc:290] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5534a1238db447bba47ff097a938c107 (127.9.117.195:45189), 34ce696027a74825a07b0ed5f2f27e0f (127.9.117.194:36887)
I20251023 20:16:57.546082 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.543946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:45189 (local address 127.9.117.193:47271)
1023 20:16:57.544020 (+ 74us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.544033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:57.544085 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:57.544337 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:57.544338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:57.544340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:57.544518 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.544521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.545279 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.545280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:57.545850 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.545852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.545927 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.545932 (+ 5us) client_negotiation.cc:769] Sending connection context
1023 20:16:57.545966 (+ 34us) client_negotiation.cc:240] Negotiation successful
1023 20:16:57.546004 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20251023 20:16:57.546489 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.544162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:47271 (local address 127.9.117.195:45189)
1023 20:16:57.544225 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.544228 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.544235 (+ 7us) server_negotiation.cc:408] Connection header received
1023 20:16:57.544261 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.544261 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.544271 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.544316 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:57.544608 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.545184 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.545937 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.546382 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.546391 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.546413 (+ 22us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.546437 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251023 20:16:57.546936 29519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9735c8d89f2438a929ce68020f7a5d3" candidate_uuid: "cf015b0894584703bfb4f916f1834684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5534a1238db447bba47ff097a938c107" is_pre_election: true
I20251023 20:16:57.547096 29519 raft_consensus.cc:2468] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf015b0894584703bfb4f916f1834684 in term 0.
I20251023 20:16:57.547209 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.543951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:36887 (local address 127.9.117.193:55073)
1023 20:16:57.544102 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.544112 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:16:57.544185 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:16:57.544599 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:16:57.544600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:16:57.544601 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:16:57.545140 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.545143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.546248 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.546250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:16:57.546958 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:16:57.546960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.547034 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.547038 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:16:57.547079 (+ 41us) client_negotiation.cc:240] Negotiation successful
1023 20:16:57.547116 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20251023 20:16:57.547367 29191 leader_election.cc:304] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5534a1238db447bba47ff097a938c107, cf015b0894584703bfb4f916f1834684; no voters:
I20251023 20:16:57.547500 29585 raft_consensus.cc:2804] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:16:57.547574 29585 raft_consensus.cc:493] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:16:57.547605 29585 raft_consensus.cc:3060] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:57.547693 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.544049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:55073 (local address 127.9.117.194:36887)
1023 20:16:57.544098 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.544100 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.544192 (+ 92us) server_negotiation.cc:408] Connection header received
1023 20:16:57.544254 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.544255 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.544271 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.544329 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:16:57.545242 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.546100 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.547048 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.547538 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.547547 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.547573 (+ 26us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.547617 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251023 20:16:57.548005 29388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9735c8d89f2438a929ce68020f7a5d3" candidate_uuid: "cf015b0894584703bfb4f916f1834684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ce696027a74825a07b0ed5f2f27e0f" is_pre_election: true
I20251023 20:16:57.548156 29388 raft_consensus.cc:2468] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf015b0894584703bfb4f916f1834684 in term 0.
I20251023 20:16:57.548449 29585 raft_consensus.cc:515] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.548594 29585 leader_election.cc:290] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [CANDIDATE]: Term 1 election: Requested vote from peers 5534a1238db447bba47ff097a938c107 (127.9.117.195:45189), 34ce696027a74825a07b0ed5f2f27e0f (127.9.117.194:36887)
I20251023 20:16:57.548794 29388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9735c8d89f2438a929ce68020f7a5d3" candidate_uuid: "cf015b0894584703bfb4f916f1834684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ce696027a74825a07b0ed5f2f27e0f"
I20251023 20:16:57.548799 29519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9735c8d89f2438a929ce68020f7a5d3" candidate_uuid: "cf015b0894584703bfb4f916f1834684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5534a1238db447bba47ff097a938c107"
I20251023 20:16:57.548880 29388 raft_consensus.cc:3060] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:57.548880 29519 raft_consensus.cc:3060] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:57.549849 29519 raft_consensus.cc:2468] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf015b0894584703bfb4f916f1834684 in term 1.
I20251023 20:16:57.549849 29388 raft_consensus.cc:2468] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf015b0894584703bfb4f916f1834684 in term 1.
I20251023 20:16:57.550060 29191 leader_election.cc:304] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5534a1238db447bba47ff097a938c107, cf015b0894584703bfb4f916f1834684; no voters:
I20251023 20:16:57.550223 29585 raft_consensus.cc:2804] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:57.550396 29585 raft_consensus.cc:697] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 1 LEADER]: Becoming Leader. State: Replica: cf015b0894584703bfb4f916f1834684, State: Running, Role: LEADER
I20251023 20:16:57.550545 29585 consensus_queue.cc:237] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } }
I20251023 20:16:57.551424 29115 catalog_manager.cc:5649] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf015b0894584703bfb4f916f1834684 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "cf015b0894584703bfb4f916f1834684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf015b0894584703bfb4f916f1834684" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 41373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 } health_report { overall_health: UNKNOWN } } }
W20251023 20:16:57.575486 29304 tablet.cc:2378] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:16:57.592224 29566 tablet.cc:2378] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:16:57.601783 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.599293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.9.117.193:41373)
1023 20:16:57.599382 (+ 89us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:57.599390 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:57.599400 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:16:57.599456 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:57.599457 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:57.599481 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:57.599579 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:16:57.599856 (+ 277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.600385 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:57.601056 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:57.601210 (+ 154us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:57.601382 (+ 172us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:16:57.601473 (+ 91us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:16:57.601567 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:57.601671 (+ 104us) server_negotiation.cc:300] Negotiation successful
1023 20:16:57.601679 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251023 20:16:57.603758 29519 raft_consensus.cc:1275] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 1 FOLLOWER]: Refusing update from remote peer cf015b0894584703bfb4f916f1834684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:57.603758 29388 raft_consensus.cc:1275] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 1 FOLLOWER]: Refusing update from remote peer cf015b0894584703bfb4f916f1834684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:16:57.604168 29585 consensus_queue.cc:1048] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5534a1238db447bba47ff097a938c107" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:57.604453 29589 consensus_queue.cc:1048] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34ce696027a74825a07b0ed5f2f27e0f" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:16:57.606686 29592 mvcc.cc:204] Tried to move back new op lower bound from 7214082529701888000 to 7214082529487757312. Current Snapshot: MvccSnapshot[applied={T|T < 7214082529701888000}]
I20251023 20:16:57.607053 29593 mvcc.cc:204] Tried to move back new op lower bound from 7214082529701888000 to 7214082529487757312. Current Snapshot: MvccSnapshot[applied={T|T < 7214082529701888000}]
I20251023 20:16:57.607581 29595 mvcc.cc:204] Tried to move back new op lower bound from 7214082529701888000 to 7214082529487757312. Current Snapshot: MvccSnapshot[applied={T|T < 7214082529701888000}]
I20251023 20:16:57.610167 29114 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59170:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:57.610268 29114 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59170:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:16:57.611191 29114 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 288c6220575b40e39a0f2e29b2f6b839: Sending DeleteTablet for 3 replicas of tablet f9735c8d89f2438a929ce68020f7a5d3
I20251023 20:16:57.611578 29237 tablet_service.cc:1552] Processing DeleteTablet for tablet f9735c8d89f2438a929ce68020f7a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:57 UTC) from {username='slave'} at 127.0.0.1:47872
I20251023 20:16:57.611608 29499 tablet_service.cc:1552] Processing DeleteTablet for tablet f9735c8d89f2438a929ce68020f7a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:57 UTC) from {username='slave'} at 127.0.0.1:46182
I20251023 20:16:57.611876 29601 tablet_replica.cc:333] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684: stopping tablet replica
I20251023 20:16:57.611858 29368 tablet_service.cc:1552] Processing DeleteTablet for tablet f9735c8d89f2438a929ce68020f7a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:16:57 UTC) from {username='slave'} at 127.0.0.1:35962
I20251023 20:16:57.612020 29601 raft_consensus.cc:2243] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:16:57.612133 29603 tablet_replica.cc:333] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: stopping tablet replica
I20251023 20:16:57.612273 29603 raft_consensus.cc:2243] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:57.612277 29601 raft_consensus.cc:2272] T f9735c8d89f2438a929ce68020f7a5d3 P cf015b0894584703bfb4f916f1834684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:57.612402 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 29175
I20251023 20:16:57.612425 29603 raft_consensus.cc:2272] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:57.613778 29603 ts_tablet_manager.cc:1916] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:57.613970 29602 tablet_replica.cc:333] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: stopping tablet replica
I20251023 20:16:57.614074 29602 raft_consensus.cc:2243] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:16:57.614193 29602 raft_consensus.cc:2272] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:16:57.614641 29602 ts_tablet_manager.cc:1916] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:16:57.616014 29602 ts_tablet_manager.cc:1929] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:57.616081 29602 log.cc:1199] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/f9735c8d89f2438a929ce68020f7a5d3
I20251023 20:16:57.616333 29603 ts_tablet_manager.cc:1929] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:16:57.616411 29602 ts_tablet_manager.cc:1950] T f9735c8d89f2438a929ce68020f7a5d3 P 5534a1238db447bba47ff097a938c107: Deleting consensus metadata
I20251023 20:16:57.616453 29603 log.cc:1199] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/f9735c8d89f2438a929ce68020f7a5d3
I20251023 20:16:57.616807 29603 ts_tablet_manager.cc:1950] T f9735c8d89f2438a929ce68020f7a5d3 P 34ce696027a74825a07b0ed5f2f27e0f: Deleting consensus metadata
I20251023 20:16:57.616950 29100 catalog_manager.cc:4985] TS 5534a1238db447bba47ff097a938c107 (127.9.117.195:45189): tablet f9735c8d89f2438a929ce68020f7a5d3 (table test-table [id=dcda1f8a0a494af080bb6e9d2967cc01]) successfully deleted
I20251023 20:16:57.617226 29102 catalog_manager.cc:4985] TS 34ce696027a74825a07b0ed5f2f27e0f (127.9.117.194:36887): tablet f9735c8d89f2438a929ce68020f7a5d3 (table test-table [id=dcda1f8a0a494af080bb6e9d2967cc01]) successfully deleted
W20251023 20:16:57.619503 29102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:41373 (error 108)
I20251023 20:16:57.619781 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 29306
I20251023 20:16:57.620426 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:57.619906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:41373 (local address 127.0.0.1:47890)
1023 20:16:57.620325 (+ 419us) negotiation.cc:107] Waiting for socket to connect
1023 20:16:57.620377 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
W20251023 20:16:57.620561 29102 catalog_manager.cc:4712] TS cf015b0894584703bfb4f916f1834684 (127.9.117.193:41373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9735c8d89f2438a929ce68020f7a5d3: Network error: Client connection negotiation failed: client connection to 127.9.117.193:41373: connect: Connection refused (error 111)
I20251023 20:16:57.625342 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 29437
I20251023 20:16:57.630455 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 29084
2025-10-23T20:16:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (872 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:16:57 dist-test-slave-x62p krb5kdc[29607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:16:57 dist-test-slave-x62p krb5kdc[29607](info): set up 2 sockets
Oct 23 20:16:57 dist-test-slave-x62p krb5kdc[29607](info): commencing operation
krb5kdc: starting...
W20251023 20:16:59.616918 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.966s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:16:59 dist-test-slave-x62p krb5kdc[29607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:16:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:16:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:59.755904 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37909
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:16:59.839327 29623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:16:59.839531 29623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:16:59.839568 29623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:16:59.841099 29623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:16:59.841153 29623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:16:59.841176 29623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:16:59.841195 29623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:16:59.841217 29623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:16:59.842890 29623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:16:59.843163 29623 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:16:59.843348 29623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:16:59.845929 29623 server_base.cc:1047] running on GCE node
W20251023 20:16:59.845903 29629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:59.845911 29631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:16:59.845922 29628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:16:59.846285 29623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:16:59.846493 29623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:16:59.847637 29623 hybrid_clock.cc:648] HybridClock initialized: now 1761250619847629 us; error 27 us; skew 500 ppm
Oct 23 20:16:59 dist-test-slave-x62p krb5kdc[29607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250619, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:16:59.850131 29623 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:16:59.850706 29623 webserver.cc:492] Webserver started at http://127.9.117.254:41273/ using document root <none> and password file <none>
I20251023 20:16:59.850893 29623 fs_manager.cc:362] Metadata directory not provided
I20251023 20:16:59.850931 29623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:16:59.851048 29623 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:16:59.852034 29623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "9a1401489792469aae13b2838413696f"
format_stamp: "Formatted at 2025-10-23 20:16:59 on dist-test-slave-x62p"
server_key: "d995083b9217f9c2c07c64ed21ee1db9"
server_key_iv: "8c0761807f3010acea69ba73f6cf39b6"
server_key_version: "encryptionkey@0"
I20251023 20:16:59.852352 29623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "9a1401489792469aae13b2838413696f"
format_stamp: "Formatted at 2025-10-23 20:16:59 on dist-test-slave-x62p"
server_key: "d995083b9217f9c2c07c64ed21ee1db9"
server_key_iv: "8c0761807f3010acea69ba73f6cf39b6"
server_key_version: "encryptionkey@0"
I20251023 20:16:59.853649 29623 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:16:59.854328 29638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:16:59.854496 29623 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:16:59.854570 29623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "9a1401489792469aae13b2838413696f"
format_stamp: "Formatted at 2025-10-23 20:16:59 on dist-test-slave-x62p"
server_key: "d995083b9217f9c2c07c64ed21ee1db9"
server_key_iv: "8c0761807f3010acea69ba73f6cf39b6"
server_key_version: "encryptionkey@0"
I20251023 20:16:59.854645 29623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:16:59.865662 29623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:16:59.867498 29623 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:16:59.867663 29623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:16:59.871850 29623 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37909
I20251023 20:16:59.871893 29690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37909 every 8 connection(s)
I20251023 20:16:59.872208 29623 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:16:59.872874 29691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:16:59.875444 29691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f: Bootstrap starting.
I20251023 20:16:59.876073 29691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f: Neither blocks nor log segments found. Creating new log.
I20251023 20:16:59.876327 29691 log.cc:826] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f: Log is configured to *not* fsync() on all Append() calls
I20251023 20:16:59.877101 29691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f: No bootstrap required, opened a new log
I20251023 20:16:59.878376 29691 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1401489792469aae13b2838413696f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37909 } }
I20251023 20:16:59.878506 29691 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:16:59.878527 29691 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1401489792469aae13b2838413696f, State: Initialized, Role: FOLLOWER
I20251023 20:16:59.878664 29691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1401489792469aae13b2838413696f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37909 } }
I20251023 20:16:59.878731 29691 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:16:59.878753 29691 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:16:59.878787 29691 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:16:59.879350 29691 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1401489792469aae13b2838413696f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37909 } }
I20251023 20:16:59.879459 29691 leader_election.cc:304] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a1401489792469aae13b2838413696f; no voters:
I20251023 20:16:59.879617 29691 leader_election.cc:290] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:16:59.879734 29694 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:16:59.879873 29691 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:16:59.879904 29694 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [term 1 LEADER]: Becoming Leader. State: Replica: 9a1401489792469aae13b2838413696f, State: Running, Role: LEADER
I20251023 20:16:59.880002 29694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1401489792469aae13b2838413696f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37909 } }
I20251023 20:16:59.880425 29695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a1401489792469aae13b2838413696f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1401489792469aae13b2838413696f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37909 } } }
I20251023 20:16:59.880504 29695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:59.880606 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 29623
I20251023 20:16:59.880751 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:16:59.880882 29696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a1401489792469aae13b2838413696f. Latest consensus state: current_term: 1 leader_uuid: "9a1401489792469aae13b2838413696f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1401489792469aae13b2838413696f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37909 } } }
I20251023 20:16:59.880951 9687 external_mini_cluster.cc:1468] Setting key f3bf2211b83dd3e8ea564ec70bc43793
I20251023 20:16:59.880954 29696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f [sys.catalog]: This master's current role is: LEADER
I20251023 20:16:59.881074 29702 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:16:59.881600 29702 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:16:59.883939 29702 catalog_manager.cc:1357] Generated new cluster ID: dd87328316d346a1931f96cff35bd6c4
I20251023 20:16:59.883993 29702 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 23 20:16:59 dist-test-slave-x62p krb5kdc[29607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:16:59.890595 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:16:59.881888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.9.117.254:37909)
1023 20:16:59.882078 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1023 20:16:59.882091 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1023 20:16:59.882122 (+ 31us) server_negotiation.cc:408] Connection header received
1023 20:16:59.882372 (+ 250us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:16:59.882378 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:16:59.882434 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:16:59.882594 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:16:59.883601 (+ 1007us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:59.884469 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:16:59.885124 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:16:59.885384 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:16:59.887488 (+ 2104us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:16:59.887494 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:16:59.887494 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:16:59.887515 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:16:59.889271 (+ 1756us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:59.889609 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:59.889612 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:59.889612 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:59.889663 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:16:59.889813 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:16:59.889814 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:16:59.889814 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:16:59.890017 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:16:59.890184 (+ 167us) server_negotiation.cc:1036] Waiting for connection context
1023 20:16:59.890266 (+ 82us) server_negotiation.cc:300] Negotiation successful
1023 20:16:59.890398 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20251023 20:16:59.892433 29702 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:16:59.892982 29702 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:16:59.900689 29702 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f: Generated new TSK 0
I20251023 20:16:59.900914 29702 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:16:59.951435 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37909
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:00.037798 29719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:00.037984 29719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:00.038009 29719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:00.039453 29719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:00.039505 29719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:00.039544 29719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:17:00.041203 29719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:00.041443 29719 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:00.041659 29719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:00.044263 29725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:00.044275 29724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:00.044283 29727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:00.044782 29719 server_base.cc:1047] running on GCE node
I20251023 20:17:00.044965 29719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:00.045188 29719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:00.046329 29719 hybrid_clock.cc:648] HybridClock initialized: now 1761250620046316 us; error 27 us; skew 500 ppm
Oct 23 20:17:00 dist-test-slave-x62p krb5kdc[29607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250620, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:00.049209 29719 init.cc:377] Logged in from keytab as kudu/127.9.117.193@KRBTEST.COM (short username kudu)
I20251023 20:17:00.049878 29719 webserver.cc:492] Webserver started at http://127.9.117.193:45767/ using document root <none> and password file <none>
I20251023 20:17:00.050091 29719 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:00.050134 29719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:00.050258 29719 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:00.051309 29719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "40d1948293314aa08074098ad0f2f227"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "86f6e400eca811d8d418096892e44dd2"
server_key_iv: "7eb6369f02793cad5e9c00ccdd99958d"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.051657 29719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "40d1948293314aa08074098ad0f2f227"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "86f6e400eca811d8d418096892e44dd2"
server_key_iv: "7eb6369f02793cad5e9c00ccdd99958d"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.052991 29719 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251023 20:17:00.053787 29734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.054041 29719 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:00.054126 29719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "40d1948293314aa08074098ad0f2f227"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "86f6e400eca811d8d418096892e44dd2"
server_key_iv: "7eb6369f02793cad5e9c00ccdd99958d"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.054193 29719 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:00.065883 29719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:00.067909 29719 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:00.068069 29719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:00.068331 29719 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:17:00.068666 29719 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:17:00.068701 29719 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.068768 29719 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:17:00.068789 29719 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.075517 29719 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:46769
I20251023 20:17:00.075558 29847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:46769 every 8 connection(s)
I20251023 20:17:00.075896 29719 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:17:00.076314 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 29719
I20251023 20:17:00.076432 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:17:00.076622 9687 external_mini_cluster.cc:1468] Setting key acdcce2ac6823bf2fe322342b8ce67f8
Oct 23 20:17:00 dist-test-slave-x62p krb5kdc[29607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250620, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.193@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.117.194@KRBTEST.COM; defaulting to no policy
I20251023 20:17:00.086388 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.076775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:34283 (local address 127.9.117.254:37909)
1023 20:17:00.076837 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.076840 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.077045 (+ 205us) server_negotiation.cc:408] Connection header received
1023 20:17:00.077648 (+ 603us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.077649 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.077667 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.077730 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:00.078528 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.079183 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.080721 (+ 1538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.080976 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.083382 (+ 2406us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:00.083387 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:00.083388 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:00.083395 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:00.084935 (+ 1540us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:00.085433 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:00.085434 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:00.085434 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:00.085456 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:00.085699 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:00.085701 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:00.085701 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:00.085832 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:00.085906 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.086222 (+ 316us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.086318 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:17:00.086457 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.076685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37909 (local address 127.9.117.193:34283)
1023 20:17:00.076954 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.076982 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:00.077538 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:00.077775 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:00.077779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:00.078122 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:00.078420 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.078424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.079273 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.079274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:00.080575 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.080601 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.080794 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.081390 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:00.081421 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:00.083289 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:00.085036 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:00.085044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:00.085048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:00.085361 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:00.085510 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:00.085511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:00.085511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:00.085574 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:00.085900 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:00.085904 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:00.086082 (+ 178us) client_negotiation.cc:769] Sending connection context
1023 20:17:00.086177 (+ 95us) client_negotiation.cc:240] Negotiation successful
1023 20:17:00.086294 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20251023 20:17:00.087044 29848 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37909
I20251023 20:17:00.087153 29848 heartbeater.cc:461] Registering TS with master...
I20251023 20:17:00.087384 29848 heartbeater.cc:507] Master 127.9.117.254:37909 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.194@KRBTEST.COM" created.
I20251023 20:17:00.088055 29655 ts_manager.cc:194] Registered new tserver with Master: 40d1948293314aa08074098ad0f2f227 (127.9.117.193:46769)
I20251023 20:17:00.089171 29655 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.193@KRBTEST.COM'} at 127.9.117.193:34283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:00.127018 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37909
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:00.217419 29855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:00.217657 29855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:00.217707 29855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:00.219254 29855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:00.219312 29855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:00.219358 29855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:17:00.221220 29855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:00.221551 29855 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:00.221774 29855 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:00.224947 29860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:00.224951 29863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:00.225229 29855 server_base.cc:1047] running on GCE node
W20251023 20:17:00.224948 29861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:00.225477 29855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:00.225720 29855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:00.226872 29855 hybrid_clock.cc:648] HybridClock initialized: now 1761250620226853 us; error 40 us; skew 500 ppm
Oct 23 20:17:00 dist-test-slave-x62p krb5kdc[29607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250620, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:00.229585 29855 init.cc:377] Logged in from keytab as kudu/127.9.117.194@KRBTEST.COM (short username kudu)
I20251023 20:17:00.230351 29855 webserver.cc:492] Webserver started at http://127.9.117.194:41983/ using document root <none> and password file <none>
I20251023 20:17:00.230611 29855 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:00.230672 29855 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:00.230814 29855 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:00.232268 29855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "cf8d0adab4cc4d469b916d76cb184912"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "9e50649321d265e02aa8fc5e71b63f31"
server_key_iv: "f413a76ff52b8042bd48ae774f23dfeb"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.232703 29855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "cf8d0adab4cc4d469b916d76cb184912"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "9e50649321d265e02aa8fc5e71b63f31"
server_key_iv: "f413a76ff52b8042bd48ae774f23dfeb"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.234225 29855 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:00.234951 29870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.235157 29855 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:00.235234 29855 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "cf8d0adab4cc4d469b916d76cb184912"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "9e50649321d265e02aa8fc5e71b63f31"
server_key_iv: "f413a76ff52b8042bd48ae774f23dfeb"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.235291 29855 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:00.253705 29855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:00.255720 29855 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:00.255843 29855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:00.256054 29855 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:17:00.256386 29855 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:17:00.256422 29855 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.256446 29855 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:17:00.256459 29855 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.262969 29855 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:42353
I20251023 20:17:00.263055 29983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:42353 every 8 connection(s)
I20251023 20:17:00.263352 29855 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
Oct 23 20:17:00 dist-test-slave-x62p krb5kdc[29607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250620, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.194@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:00.272154 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.264140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:58769 (local address 127.9.117.254:37909)
1023 20:17:00.264207 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.264210 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.264358 (+ 148us) server_negotiation.cc:408] Connection header received
1023 20:17:00.264954 (+ 596us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.264956 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.264969 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.265031 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:00.265851 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.266362 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.267405 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.267601 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.269622 (+ 2021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:00.269626 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:00.269627 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:00.269633 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:00.270896 (+ 1263us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:00.271277 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:00.271278 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:00.271279 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:00.271308 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:00.271485 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:00.271486 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:00.271487 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:00.271601 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:00.271665 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.271937 (+ 272us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.272053 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251023 20:17:00.272264 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.264093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37909 (local address 127.9.117.194:58769)
1023 20:17:00.264249 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.264275 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:00.264858 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:00.265100 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:00.265104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:00.265444 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:00.265747 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.265750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.266444 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.266447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:00.267321 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.267323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.267470 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.268060 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:00.268081 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:00.269523 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:00.270982 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:00.270989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:00.270992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:00.271208 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:00.271364 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:00.271364 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:00.271365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:00.271429 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:00.271657 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:00.271662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:00.271807 (+ 145us) client_negotiation.cc:769] Sending connection context
1023 20:17:00.271911 (+ 104us) client_negotiation.cc:240] Negotiation successful
1023 20:17:00.272060 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20251023 20:17:00.272341 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 29855
I20251023 20:17:00.272461 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:17:00.272665 9687 external_mini_cluster.cc:1468] Setting key b47a4eb90bf84fca0082d6745b9c151b
I20251023 20:17:00.272742 29984 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37909
I20251023 20:17:00.272855 29984 heartbeater.cc:461] Registering TS with master...
I20251023 20:17:00.273090 29984 heartbeater.cc:507] Master 127.9.117.254:37909 requested a full tablet report, sending...
I20251023 20:17:00.273456 29655 ts_manager.cc:194] Registered new tserver with Master: cf8d0adab4cc4d469b916d76cb184912 (127.9.117.194:42353)
I20251023 20:17:00.273861 29655 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.194@KRBTEST.COM'} at 127.9.117.194:58769
WARNING: no policy specified for kudu/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:00.317848 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:37909
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:00.404558 29991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:00.404840 29991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:00.404878 29991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:00.406589 29991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:00.406659 29991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:00.406694 29991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:17:00.408583 29991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.9.117.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:00.408918 29991 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:00.409109 29991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:00.412249 29997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:00.412230 29999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:00.412223 29996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:00.412467 29991 server_base.cc:1047] running on GCE node
I20251023 20:17:00.412791 29991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:00.413048 29991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:00.414208 29991 hybrid_clock.cc:648] HybridClock initialized: now 1761250620414177 us; error 41 us; skew 500 ppm
Oct 23 20:17:00 dist-test-slave-x62p krb5kdc[29607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250620, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:00.416957 29991 init.cc:377] Logged in from keytab as kudu/127.9.117.195@KRBTEST.COM (short username kudu)
I20251023 20:17:00.417601 29991 webserver.cc:492] Webserver started at http://127.9.117.195:35035/ using document root <none> and password file <none>
I20251023 20:17:00.417826 29991 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:00.417870 29991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:00.417999 29991 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:00.419104 29991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "bc12c04795f94e0f920271ceec5fc0bc"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "54a0e7864ca4740bc27fddb0b14e0d2e"
server_key_iv: "d896f8a292f018ab13f6dad3067babd0"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.419452 29991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "bc12c04795f94e0f920271ceec5fc0bc"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "54a0e7864ca4740bc27fddb0b14e0d2e"
server_key_iv: "d896f8a292f018ab13f6dad3067babd0"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.420845 29991 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:17:00.421666 30006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.421890 29991 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:00.421967 29991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "bc12c04795f94e0f920271ceec5fc0bc"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "54a0e7864ca4740bc27fddb0b14e0d2e"
server_key_iv: "d896f8a292f018ab13f6dad3067babd0"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.422036 29991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:00.445143 29991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:00.447407 29991 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:00.447572 29991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:00.447834 29991 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:17:00.448179 29991 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:17:00.448213 29991 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.448251 29991 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:17:00.448278 29991 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.454819 29991 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:32863
I20251023 20:17:00.454874 30119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:32863 every 8 connection(s)
I20251023 20:17:00.455205 29991 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
Oct 23 20:17:00 dist-test-slave-x62p krb5kdc[29607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250620, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.195@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:00.463224 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 29991
I20251023 20:17:00.463339 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:17:00.463563 9687 external_mini_cluster.cc:1468] Setting key 7e8acdac668e5e21e855f79a9b642704
I20251023 20:17:00.464954 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.455948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:47881 (local address 127.9.117.254:37909)
1023 20:17:00.456012 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.456015 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.456206 (+ 191us) server_negotiation.cc:408] Connection header received
1023 20:17:00.456730 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.456732 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.456791 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.456859 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:00.457739 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.458317 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.459473 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.459643 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.461831 (+ 2188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:00.461836 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:00.461836 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:00.461844 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:00.463305 (+ 1461us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:00.463878 (+ 573us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:00.463879 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:00.463879 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:00.463910 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:00.464146 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:00.464148 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:00.464148 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:00.464303 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:00.464364 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.464778 (+ 414us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.464884 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251023 20:17:00.465051 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.455911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37909 (local address 127.9.117.195:47881)
1023 20:17:00.456069 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.456101 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:00.456622 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:00.456923 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:00.456927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:00.457282 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:00.457620 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.457624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.458403 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.458404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:00.459355 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.459358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.459504 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.460124 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:00.460148 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:00.461706 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:00.463424 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:00.463434 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:00.463438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:00.463781 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:00.463971 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:00.463972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:00.463973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:00.464078 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:00.464367 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:00.464373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:00.464538 (+ 165us) client_negotiation.cc:769] Sending connection context
1023 20:17:00.464674 (+ 136us) client_negotiation.cc:240] Negotiation successful
1023 20:17:00.464877 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20251023 20:17:00.465572 30120 heartbeater.cc:344] Connected to a master server at 127.9.117.254:37909
I20251023 20:17:00.465669 30120 heartbeater.cc:461] Registering TS with master...
I20251023 20:17:00.465888 30120 heartbeater.cc:507] Master 127.9.117.254:37909 requested a full tablet report, sending...
I20251023 20:17:00.466315 29655 ts_manager.cc:194] Registered new tserver with Master: bc12c04795f94e0f920271ceec5fc0bc (127.9.117.195:32863)
I20251023 20:17:00.466688 29655 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.117.195@KRBTEST.COM'} at 127.9.117.195:47881
I20251023 20:17:00.475860 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:17:00.483523 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.477395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.9.117.254:37909)
1023 20:17:00.477477 (+ 82us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.477480 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.477571 (+ 91us) server_negotiation.cc:408] Connection header received
1023 20:17:00.477623 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.477624 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.477639 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.477708 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:00.478493 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.479014 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.479843 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.479993 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.481194 (+ 1201us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:00.481199 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:00.481199 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:00.481207 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:00.482561 (+ 1354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:00.482889 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:00.482890 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:00.482890 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:00.482913 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:00.483028 (+ 115us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:00.483029 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:00.483029 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:00.483164 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:00.483250 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.483369 (+ 119us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.483446 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251023 20:17:00.485147 29655 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:17:00.486517 29655 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:17:00.492589 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.489193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:46769 (local address 127.0.0.1:41026)
1023 20:17:00.489429 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.489464 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:00.489548 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:00.490190 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:00.490190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:00.490192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:00.490504 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.490505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.491457 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.491459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:00.492253 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.492255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.492324 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.492383 (+ 59us) client_negotiation.cc:769] Sending connection context
1023 20:17:00.492439 (+ 56us) client_negotiation.cc:240] Negotiation successful
1023 20:17:00.492487 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20251023 20:17:00.493085 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.489250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42353 (local address 127.0.0.1:45828)
1023 20:17:00.489588 (+ 338us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.489601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:00.489867 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:00.490153 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:00.490156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:00.490158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:00.490504 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.490505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.491876 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.491877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:00.492868 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.492872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.492939 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.492943 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:17:00.492974 (+ 31us) client_negotiation.cc:240] Negotiation successful
1023 20:17:00.493015 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":97,"threads_started":1}
I20251023 20:17:00.493232 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.489392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.9.117.193:46769)
1023 20:17:00.490024 (+ 632us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.490032 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.490041 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:17:00.490085 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.490087 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.490106 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.490180 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:00.490605 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.491362 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.492368 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.492977 (+ 609us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.493028 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.493096 (+ 68us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.493134 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":113,"threads_started":1}
I20251023 20:17:00.493830 29782 tablet_service.cc:1505] Processing CreateTablet for tablet ff93da001008410e9aaef5acc87a658c (DEFAULT_TABLE table=test-table [id=220367d5a2594724bec1f694696b23e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:17:00.494107 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.489380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.9.117.194:42353)
1023 20:17:00.489599 (+ 219us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.489608 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.489857 (+ 249us) server_negotiation.cc:408] Connection header received
1023 20:17:00.489945 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.489947 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.489967 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.490034 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:00.490610 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.491776 (+ 1166us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.493295 (+ 1519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.493895 (+ 600us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.493949 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.493982 (+ 33us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.494030 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20251023 20:17:00.494163 29782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff93da001008410e9aaef5acc87a658c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:00.494331 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.489656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.9.117.195:32863)
1023 20:17:00.489832 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.489841 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.489853 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:17:00.489900 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.489903 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.489923 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.490000 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:00.490513 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.491755 (+ 1242us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.493543 (+ 1788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.494112 (+ 569us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.494160 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.494195 (+ 35us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.494252 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20251023 20:17:00.494561 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.489407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:32863 (local address 127.0.0.1:37154)
1023 20:17:00.489637 (+ 230us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.489648 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:00.489690 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:00.490132 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:00.490135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:00.490137 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:00.490422 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.490426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.491875 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.491876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:00.493172 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.493176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.493256 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.493260 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:17:00.494443 (+ 1183us) client_negotiation.cc:240] Negotiation successful
1023 20:17:00.494500 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
I20251023 20:17:00.494695 29918 tablet_service.cc:1505] Processing CreateTablet for tablet ff93da001008410e9aaef5acc87a658c (DEFAULT_TABLE table=test-table [id=220367d5a2594724bec1f694696b23e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:17:00.495013 29918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff93da001008410e9aaef5acc87a658c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:00.495204 30054 tablet_service.cc:1505] Processing CreateTablet for tablet ff93da001008410e9aaef5acc87a658c (DEFAULT_TABLE table=test-table [id=220367d5a2594724bec1f694696b23e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:17:00.495522 30054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff93da001008410e9aaef5acc87a658c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:00.497608 30137 tablet_bootstrap.cc:492] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227: Bootstrap starting.
I20251023 20:17:00.498225 30137 tablet_bootstrap.cc:654] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:00.498512 30137 log.cc:826] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:00.498509 30139 tablet_bootstrap.cc:492] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Bootstrap starting.
I20251023 20:17:00.498903 30138 tablet_bootstrap.cc:492] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912: Bootstrap starting.
I20251023 20:17:00.499209 30139 tablet_bootstrap.cc:654] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:00.499251 30137 tablet_bootstrap.cc:492] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227: No bootstrap required, opened a new log
I20251023 20:17:00.499320 30137 ts_tablet_manager.cc:1403] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:17:00.499545 30138 tablet_bootstrap.cc:654] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:00.499543 30139 log.cc:826] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:00.499866 30138 log.cc:826] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:00.500327 30139 tablet_bootstrap.cc:492] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: No bootstrap required, opened a new log
I20251023 20:17:00.500453 30139 ts_tablet_manager.cc:1403] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:17:00.500875 30138 tablet_bootstrap.cc:492] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912: No bootstrap required, opened a new log
I20251023 20:17:00.500949 30138 ts_tablet_manager.cc:1403] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:17:00.501291 30137 raft_consensus.cc:359] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.501422 30137 raft_consensus.cc:385] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:00.501458 30137 raft_consensus.cc:740] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d1948293314aa08074098ad0f2f227, State: Initialized, Role: FOLLOWER
I20251023 20:17:00.501588 30137 consensus_queue.cc:260] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.501850 30137 ts_tablet_manager.cc:1434] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:17:00.501983 29848 heartbeater.cc:499] Master 127.9.117.254:37909 was elected leader, sending a full tablet report...
I20251023 20:17:00.502755 30139 raft_consensus.cc:359] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.502909 30139 raft_consensus.cc:385] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:00.502938 30139 raft_consensus.cc:740] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc12c04795f94e0f920271ceec5fc0bc, State: Initialized, Role: FOLLOWER
I20251023 20:17:00.503041 30139 consensus_queue.cc:260] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.503201 30138 raft_consensus.cc:359] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.503361 30138 raft_consensus.cc:385] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:00.503376 30139 ts_tablet_manager.cc:1434] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:17:00.503392 30138 raft_consensus.cc:740] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8d0adab4cc4d469b916d76cb184912, State: Initialized, Role: FOLLOWER
I20251023 20:17:00.503418 30120 heartbeater.cc:499] Master 127.9.117.254:37909 was elected leader, sending a full tablet report...
I20251023 20:17:00.503505 30138 consensus_queue.cc:260] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.503746 30138 ts_tablet_manager.cc:1434] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:17:00.503785 29984 heartbeater.cc:499] Master 127.9.117.254:37909 was elected leader, sending a full tablet report...
W20251023 20:17:00.514000 29985 tablet.cc:2378] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251023 20:17:00.576793 29849 tablet.cc:2378] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:17:00.691221 30143 raft_consensus.cc:493] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:17:00.691365 30143 raft_consensus.cc:515] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.691766 30143 leader_election.cc:290] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc12c04795f94e0f920271ceec5fc0bc (127.9.117.195:32863), cf8d0adab4cc4d469b916d76cb184912 (127.9.117.194:42353)
I20251023 20:17:00.694315 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.691953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:32863 (local address 127.9.117.193:47849)
1023 20:17:00.692140 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.692151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:00.692224 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:00.692387 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:00.692388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:00.692391 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:00.692666 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.692668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.693398 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.693400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:00.694100 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.694103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.694178 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.694182 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:17:00.694207 (+ 25us) client_negotiation.cc:240] Negotiation successful
1023 20:17:00.694237 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20251023 20:17:00.694486 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.692000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42353 (local address 127.9.117.193:42233)
1023 20:17:00.692162 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.692171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:00.692254 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:00.692414 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:00.692415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:00.692417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:00.692747 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.692750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.693479 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.693480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:00.694248 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:00.694250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.694328 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.694331 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:17:00.694364 (+ 33us) client_negotiation.cc:240] Negotiation successful
1023 20:17:00.694401 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20251023 20:17:00.694872 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.692054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:47849 (local address 127.9.117.195:32863)
1023 20:17:00.692109 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.692112 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.692230 (+ 118us) server_negotiation.cc:408] Connection header received
1023 20:17:00.692289 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.692291 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.692308 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.692364 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:00.692788 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.693306 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.694193 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.694735 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.694745 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.694769 (+ 24us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.694799 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:17:00.694967 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.692090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:42233 (local address 127.9.117.194:42353)
1023 20:17:00.692138 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.692140 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.692256 (+ 116us) server_negotiation.cc:408] Connection header received
1023 20:17:00.692321 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.692322 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.692334 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.692397 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:00.692851 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.693385 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.694347 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.694854 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.694861 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.694881 (+ 20us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.694904 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:17:00.695246 29938 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff93da001008410e9aaef5acc87a658c" candidate_uuid: "40d1948293314aa08074098ad0f2f227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8d0adab4cc4d469b916d76cb184912" is_pre_election: true
I20251023 20:17:00.695232 30074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff93da001008410e9aaef5acc87a658c" candidate_uuid: "40d1948293314aa08074098ad0f2f227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc12c04795f94e0f920271ceec5fc0bc" is_pre_election: true
I20251023 20:17:00.695376 30074 raft_consensus.cc:2468] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d1948293314aa08074098ad0f2f227 in term 0.
I20251023 20:17:00.695389 29938 raft_consensus.cc:2468] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d1948293314aa08074098ad0f2f227 in term 0.
I20251023 20:17:00.695609 29736 leader_election.cc:304] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d1948293314aa08074098ad0f2f227, cf8d0adab4cc4d469b916d76cb184912; no voters:
I20251023 20:17:00.695783 30143 raft_consensus.cc:2804] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:17:00.695852 30143 raft_consensus.cc:493] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:17:00.695873 30143 raft_consensus.cc:3060] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:00.696547 30143 raft_consensus.cc:515] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.696677 30143 leader_election.cc:290] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [CANDIDATE]: Term 1 election: Requested vote from peers bc12c04795f94e0f920271ceec5fc0bc (127.9.117.195:32863), cf8d0adab4cc4d469b916d76cb184912 (127.9.117.194:42353)
I20251023 20:17:00.696925 30074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff93da001008410e9aaef5acc87a658c" candidate_uuid: "40d1948293314aa08074098ad0f2f227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc12c04795f94e0f920271ceec5fc0bc"
I20251023 20:17:00.696933 29938 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff93da001008410e9aaef5acc87a658c" candidate_uuid: "40d1948293314aa08074098ad0f2f227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8d0adab4cc4d469b916d76cb184912"
I20251023 20:17:00.697016 29938 raft_consensus.cc:3060] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:00.697018 30074 raft_consensus.cc:3060] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:00.697772 29938 raft_consensus.cc:2468] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d1948293314aa08074098ad0f2f227 in term 1.
I20251023 20:17:00.697768 30074 raft_consensus.cc:2468] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d1948293314aa08074098ad0f2f227 in term 1.
I20251023 20:17:00.697948 29736 leader_election.cc:304] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d1948293314aa08074098ad0f2f227, cf8d0adab4cc4d469b916d76cb184912; no voters:
I20251023 20:17:00.698055 30143 raft_consensus.cc:2804] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:00.698235 30143 raft_consensus.cc:697] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 1 LEADER]: Becoming Leader. State: Replica: 40d1948293314aa08074098ad0f2f227, State: Running, Role: LEADER
I20251023 20:17:00.698334 30143 consensus_queue.cc:237] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } }
I20251023 20:17:00.699113 29654 catalog_manager.cc:5649] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40d1948293314aa08074098ad0f2f227 (127.9.117.193). New cstate: current_term: 1 leader_uuid: "40d1948293314aa08074098ad0f2f227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1948293314aa08074098ad0f2f227" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 46769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 } health_report { overall_health: UNKNOWN } } }
W20251023 20:17:00.705890 30121 tablet.cc:2378] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:17:00.720952 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.718881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.9.117.193:46769)
1023 20:17:00.718951 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.718954 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.718963 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:17:00.719009 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.719010 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.719032 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.719085 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:00.719339 (+ 254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.719857 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.720401 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.720561 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.720603 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:00.720747 (+ 144us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:00.720834 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.720855 (+ 21us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.720888 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251023 20:17:00.722791 30074 raft_consensus.cc:1275] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 1 FOLLOWER]: Refusing update from remote peer 40d1948293314aa08074098ad0f2f227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:17:00.722777 29938 raft_consensus.cc:1275] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 1 FOLLOWER]: Refusing update from remote peer 40d1948293314aa08074098ad0f2f227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:17:00.723181 30148 consensus_queue.cc:1048] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc12c04795f94e0f920271ceec5fc0bc" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:17:00.723320 30143 consensus_queue.cc:1048] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf8d0adab4cc4d469b916d76cb184912" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:17:00.725950 30151 mvcc.cc:204] Tried to move back new op lower bound from 7214082542477594624 to 7214082542380937216. Current Snapshot: MvccSnapshot[applied={T|T < 7214082542477594624}]
I20251023 20:17:00.726172 30152 mvcc.cc:204] Tried to move back new op lower bound from 7214082542477594624 to 7214082542380937216. Current Snapshot: MvccSnapshot[applied={T|T < 7214082542477594624}]
I20251023 20:17:00.726215 30153 mvcc.cc:204] Tried to move back new op lower bound from 7214082542477594624 to 7214082542380937216. Current Snapshot: MvccSnapshot[applied={T|T < 7214082542477594624}]
I20251023 20:17:00.728677 29655 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54646:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:17:00.728824 29655 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54646:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:17:00.729616 29655 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 9a1401489792469aae13b2838413696f: Sending DeleteTablet for 3 replicas of tablet ff93da001008410e9aaef5acc87a658c
I20251023 20:17:00.729983 29782 tablet_service.cc:1552] Processing DeleteTablet for tablet ff93da001008410e9aaef5acc87a658c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:17:00 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:41026
I20251023 20:17:00.730021 30054 tablet_service.cc:1552] Processing DeleteTablet for tablet ff93da001008410e9aaef5acc87a658c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:17:00 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:37154
I20251023 20:17:00.730260 30161 tablet_replica.cc:333] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: stopping tablet replica
I20251023 20:17:00.730305 29918 tablet_service.cc:1552] Processing DeleteTablet for tablet ff93da001008410e9aaef5acc87a658c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:17:00 UTC) from {username='kudu', principal='kudu/127.9.117.254@KRBTEST.COM'} at 127.0.0.1:45828
I20251023 20:17:00.730401 30161 raft_consensus.cc:2243] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:17:00.730405 30162 tablet_replica.cc:333] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227: stopping tablet replica
I20251023 20:17:00.730492 30162 raft_consensus.cc:2243] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:17:00.730508 30161 raft_consensus.cc:2272] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:17:00.730623 30162 raft_consensus.cc:2272] T ff93da001008410e9aaef5acc87a658c P 40d1948293314aa08074098ad0f2f227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:17:00.730885 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 29719
I20251023 20:17:00.731684 30161 ts_tablet_manager.cc:1916] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:17:00.733165 30161 ts_tablet_manager.cc:1929] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:17:00.733240 30161 log.cc:1199] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/ff93da001008410e9aaef5acc87a658c
I20251023 20:17:00.733472 30161 ts_tablet_manager.cc:1950] T ff93da001008410e9aaef5acc87a658c P bc12c04795f94e0f920271ceec5fc0bc: Deleting consensus metadata
I20251023 20:17:00.733942 29641 catalog_manager.cc:4985] TS bc12c04795f94e0f920271ceec5fc0bc (127.9.117.195:32863): tablet ff93da001008410e9aaef5acc87a658c (table test-table [id=220367d5a2594724bec1f694696b23e6]) successfully deleted
W20251023 20:17:00.736907 29641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:46769 (error 108)
I20251023 20:17:00.737061 30163 tablet_replica.cc:333] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912: stopping tablet replica
I20251023 20:17:00.737185 30163 raft_consensus.cc:2243] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:17:00.737334 30163 raft_consensus.cc:2272] T ff93da001008410e9aaef5acc87a658c P cf8d0adab4cc4d469b916d76cb184912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:17:00.737618 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 29855
I20251023 20:17:00.737679 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.737389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:46769 (local address 127.0.0.1:41038)
1023 20:17:00.737480 (+ 91us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.737543 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20251023 20:17:00.738379 29641 catalog_manager.cc:4712] TS 40d1948293314aa08074098ad0f2f227 (127.9.117.193:46769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff93da001008410e9aaef5acc87a658c: Network error: Client connection negotiation failed: client connection to 127.9.117.193:46769: connect: Connection refused (error 111)
I20251023 20:17:00.743445 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 29991
I20251023 20:17:00.743786 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.743482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:42353 (local address 127.0.0.1:45830)
1023 20:17:00.743681 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:00.743727 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.194:42353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20251023 20:17:00.748641 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 29623
2025-10-23T20:17:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3119 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2025-10-23T20:17:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:00Z Disabled control of system clock
I20251023 20:17:00.777179 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:35135
--webserver_interface=127.9.117.254
--webserver_port=0
--builtin_ntp_servers=127.9.117.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251023 20:17:00.862839 30168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:00.863032 30168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:00.863054 30168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:00.864589 30168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:00.864651 30168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:00.864665 30168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:00.864677 30168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:00.864689 30168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:00.866358 30168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:35135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:00.866652 30168 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:00.866923 30168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:00.869498 30176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:00.869500 30173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:00.869531 30174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:00.869765 30168 server_base.cc:1047] running on GCE node
I20251023 20:17:00.870003 30168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:00.870256 30168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:00.871415 30168 hybrid_clock.cc:648] HybridClock initialized: now 1761250620871398 us; error 32 us; skew 500 ppm
I20251023 20:17:00.872575 30168 webserver.cc:492] Webserver started at http://127.9.117.254:44195/ using document root <none> and password file <none>
I20251023 20:17:00.872845 30168 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:00.872896 30168 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:00.873021 30168 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:00.874042 30168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "78976784376647f79d43d71bc212ddd3"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "e7734ea9f0b9cece5be02fa303dfadfb"
server_key_iv: "2192c53f8bae85a1676d7774e09dc939"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.874365 30168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "78976784376647f79d43d71bc212ddd3"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "e7734ea9f0b9cece5be02fa303dfadfb"
server_key_iv: "2192c53f8bae85a1676d7774e09dc939"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.875576 30168 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:17:00.876266 30182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:00.876447 30168 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:17:00.876523 30168 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "78976784376647f79d43d71bc212ddd3"
format_stamp: "Formatted at 2025-10-23 20:17:00 on dist-test-slave-x62p"
server_key: "e7734ea9f0b9cece5be02fa303dfadfb"
server_key_iv: "2192c53f8bae85a1676d7774e09dc939"
server_key_version: "encryptionkey@0"
I20251023 20:17:00.876595 30168 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:00.887079 30168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:00.887413 30168 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:00.887540 30168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:00.891880 30168 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:35135
I20251023 20:17:00.891927 30234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:35135 every 8 connection(s)
I20251023 20:17:00.892251 30168 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:00.892884 30235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:00.895475 30235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3: Bootstrap starting.
I20251023 20:17:00.896090 30235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:00.896349 30235 log.cc:826] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:00.897190 30235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3: No bootstrap required, opened a new log
I20251023 20:17:00.898448 30235 raft_consensus.cc:359] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78976784376647f79d43d71bc212ddd3" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35135 } }
I20251023 20:17:00.898602 30235 raft_consensus.cc:385] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:00.898639 30235 raft_consensus.cc:740] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78976784376647f79d43d71bc212ddd3, State: Initialized, Role: FOLLOWER
I20251023 20:17:00.898778 30235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78976784376647f79d43d71bc212ddd3" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35135 } }
I20251023 20:17:00.898837 30235 raft_consensus.cc:399] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:00.898877 30235 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:00.898923 30235 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:00.899497 30235 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78976784376647f79d43d71bc212ddd3" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35135 } }
I20251023 20:17:00.899616 30235 leader_election.cc:304] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78976784376647f79d43d71bc212ddd3; no voters:
I20251023 20:17:00.899775 30235 leader_election.cc:290] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:00.899870 30238 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:00.900010 30235 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:00.900020 30238 raft_consensus.cc:697] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [term 1 LEADER]: Becoming Leader. State: Replica: 78976784376647f79d43d71bc212ddd3, State: Running, Role: LEADER
I20251023 20:17:00.900146 30238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78976784376647f79d43d71bc212ddd3" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35135 } }
I20251023 20:17:00.900511 30239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78976784376647f79d43d71bc212ddd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78976784376647f79d43d71bc212ddd3" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35135 } } }
I20251023 20:17:00.900571 30240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78976784376647f79d43d71bc212ddd3. Latest consensus state: current_term: 1 leader_uuid: "78976784376647f79d43d71bc212ddd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78976784376647f79d43d71bc212ddd3" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 35135 } } }
I20251023 20:17:00.900599 30239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:00.900799 30240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:00.901078 30246 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:00.901650 30246 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:00.902033 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 30168
I20251023 20:17:00.902143 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:00.902334 9687 external_mini_cluster.cc:1468] Setting key cd596483da93e4e471ca058929f587d1
I20251023 20:17:00.903595 30246 catalog_manager.cc:1357] Generated new cluster ID: e350e47755454220ac5c1a7874944279
I20251023 20:17:00.903645 30246 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:00.907439 30246 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:00.908154 30246 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:00.909597 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:00.903057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.9.117.254:35135)
1023 20:17:00.903706 (+ 649us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:00.903713 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:00.903736 (+ 23us) server_negotiation.cc:408] Connection header received
1023 20:17:00.903853 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:00.903859 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:00.903915 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:00.904534 (+ 619us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:00.904667 (+ 133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.906074 (+ 1407us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:00.907070 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:00.907302 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:00.908970 (+ 1668us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:00.908975 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:00.908976 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:00.909174 (+ 198us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:00.909202 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:00.909210 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:00.909282 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:00.909374 (+ 92us) server_negotiation.cc:300] Negotiation successful
1023 20:17:00.909458 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":62,"threads_started":1}
I20251023 20:17:00.918114 30246 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3: Generated new TSK 0
I20251023 20:17:00.918367 30246 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:00.920127 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.193:0
--local_ip_for_outbound_sockets=127.9.117.193
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:35135
--builtin_ntp_servers=127.9.117.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251023 20:17:01.009464 30259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:01.009680 30259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:01.009706 30259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:01.011485 30259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:01.011561 30259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:01.011598 30259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.193
I20251023 20:17:01.013399 30259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.117.193
--webserver_port=0
--tserver_master_addrs=127.9.117.254:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.193
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:01.013695 30259 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:01.013963 30259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:01.017006 30265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:01.017452 30267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:01.017508 30259 server_base.cc:1047] running on GCE node
W20251023 20:17:01.017453 30264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:01.018021 30259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:01.018368 30259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:01.019579 30259 hybrid_clock.cc:648] HybridClock initialized: now 1761250621019559 us; error 36 us; skew 500 ppm
I20251023 20:17:01.021095 30259 webserver.cc:492] Webserver started at http://127.9.117.193:34583/ using document root <none> and password file <none>
I20251023 20:17:01.021327 30259 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:01.021373 30259 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:01.021488 30259 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:01.022567 30259 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/data/instance:
uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "196167961bd7ce6fd8097fec2094dfad"
server_key_iv: "e543f9b40eed70c04571764c0c6d87ad"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.022897 30259 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance:
uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "196167961bd7ce6fd8097fec2094dfad"
server_key_iv: "e543f9b40eed70c04571764c0c6d87ad"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.024263 30259 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:17:01.025099 30273 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.025295 30259 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:17:01.025388 30259 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/wal
uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "196167961bd7ce6fd8097fec2094dfad"
server_key_iv: "e543f9b40eed70c04571764c0c6d87ad"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.025475 30259 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:01.044142 30259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:01.044564 30259 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:01.044713 30259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:01.045043 30259 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:17:01.045466 30259 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:17:01.045517 30259 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.045558 30259 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:17:01.045583 30259 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.053771 30259 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.193:45957
I20251023 20:17:01.053838 30386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.193:45957 every 8 connection(s)
I20251023 20:17:01.054239 30259 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/data/info.pb
I20251023 20:17:01.055717 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 30259
I20251023 20:17:01.055823 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-0/wal/instance
I20251023 20:17:01.056022 9687 external_mini_cluster.cc:1468] Setting key 334b4dbc31fde445f22355c60abef587
I20251023 20:17:01.057044 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.194:0
--local_ip_for_outbound_sockets=127.9.117.194
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:35135
--builtin_ntp_servers=127.9.117.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251023 20:17:01.063010 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.055114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.193:38923 (local address 127.9.117.254:35135)
1023 20:17:01.055173 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.055175 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.055376 (+ 201us) server_negotiation.cc:408] Connection header received
1023 20:17:01.056115 (+ 739us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.056117 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.056134 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.056210 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:01.057050 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.057909 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.058768 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.059000 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.062595 (+ 3595us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:01.062604 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:01.062605 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:01.062613 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:01.062635 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:01.062640 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:01.062712 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.062829 (+ 117us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.062883 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:17:01.063365 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.055011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:35135 (local address 127.9.117.193:38923)
1023 20:17:01.055254 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.055287 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:01.055987 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:01.056404 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:01.056409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:01.056420 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:01.056942 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.056947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.058025 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.058026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:01.058658 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.058661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.058809 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.062416 (+ 3607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:17:01.062417 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:01.062456 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:17:01.062483 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:01.062723 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:01.062729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:01.062733 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:17:01.062819 (+ 86us) client_negotiation.cc:240] Negotiation successful
1023 20:17:01.062940 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20251023 20:17:01.064180 30387 heartbeater.cc:344] Connected to a master server at 127.9.117.254:35135
I20251023 20:17:01.064306 30387 heartbeater.cc:461] Registering TS with master...
I20251023 20:17:01.064618 30387 heartbeater.cc:507] Master 127.9.117.254:35135 requested a full tablet report, sending...
I20251023 20:17:01.065281 30199 ts_manager.cc:194] Registered new tserver with Master: 0e1ad0e9eb694acf8ed00a8ed73f3156 (127.9.117.193:45957)
I20251023 20:17:01.066322 30199 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.193:38923
W20251023 20:17:01.141682 30390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:01.141860 30390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:01.141884 30390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:01.143283 30390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:01.143321 30390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:01.143347 30390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.194
I20251023 20:17:01.144843 30390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.117.194
--webserver_port=0
--tserver_master_addrs=127.9.117.254:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.194
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:01.145045 30390 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:01.145242 30390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:01.147967 30398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:01.147979 30396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:01.148025 30395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:01.148272 30390 server_base.cc:1047] running on GCE node
I20251023 20:17:01.148434 30390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:01.148665 30390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:01.149806 30390 hybrid_clock.cc:648] HybridClock initialized: now 1761250621149805 us; error 40 us; skew 500 ppm
I20251023 20:17:01.150985 30390 webserver.cc:492] Webserver started at http://127.9.117.194:38147/ using document root <none> and password file <none>
I20251023 20:17:01.151185 30390 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:01.151229 30390 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:01.151357 30390 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:01.152364 30390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/data/instance:
uuid: "ca994b55b43f415fae40a06233198f25"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "2dfe7e1d7c769d67d8064680b67560d6"
server_key_iv: "efc0b828d86111c94a2c329f12579a39"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.152689 30390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance:
uuid: "ca994b55b43f415fae40a06233198f25"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "2dfe7e1d7c769d67d8064680b67560d6"
server_key_iv: "efc0b828d86111c94a2c329f12579a39"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.153932 30390 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:01.154605 30404 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.154798 30390 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:17:01.154873 30390 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/wal
uuid: "ca994b55b43f415fae40a06233198f25"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "2dfe7e1d7c769d67d8064680b67560d6"
server_key_iv: "efc0b828d86111c94a2c329f12579a39"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.154938 30390 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:01.171334 30390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:01.171711 30390 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:01.171849 30390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:01.172080 30390 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:17:01.172418 30390 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:17:01.172453 30390 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.172492 30390 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:17:01.172518 30390 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.178813 30390 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.194:36997
I20251023 20:17:01.178908 30518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.194:36997 every 8 connection(s)
I20251023 20:17:01.179191 30390 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/data/info.pb
I20251023 20:17:01.182379 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 30390
I20251023 20:17:01.182499 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/wal/instance
I20251023 20:17:01.182722 9687 external_mini_cluster.cc:1468] Setting key 07d45437565cb74df22c6caa9c5f4afc
I20251023 20:17:01.184079 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.117.195:0
--local_ip_for_outbound_sockets=127.9.117.195
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:35135
--builtin_ntp_servers=127.9.117.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251023 20:17:01.184312 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.179974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.194:60599 (local address 127.9.117.254:35135)
1023 20:17:01.180045 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.180048 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.180254 (+ 206us) server_negotiation.cc:408] Connection header received
1023 20:17:01.180931 (+ 677us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.180932 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.180954 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.181020 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:01.181527 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.182055 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.182930 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.183110 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.183951 (+ 841us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:01.183955 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:01.183955 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:01.183960 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:01.183978 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:01.183982 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:01.184029 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.184151 (+ 122us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.184197 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251023 20:17:01.184566 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.179941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:35135 (local address 127.9.117.194:60599)
1023 20:17:01.180148 (+ 207us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.180175 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:01.180824 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:01.181077 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:01.181081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:01.181091 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:01.181435 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.181438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.182165 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.182166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:01.182839 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.182842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.183006 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.183785 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:17:01.183786 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:01.183827 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:17:01.183856 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:01.184053 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:01.184058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:01.184061 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:17:01.184149 (+ 88us) client_negotiation.cc:240] Negotiation successful
1023 20:17:01.184264 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20251023 20:17:01.185252 30519 heartbeater.cc:344] Connected to a master server at 127.9.117.254:35135
I20251023 20:17:01.185384 30519 heartbeater.cc:461] Registering TS with master...
I20251023 20:17:01.185745 30519 heartbeater.cc:507] Master 127.9.117.254:35135 requested a full tablet report, sending...
I20251023 20:17:01.186246 30199 ts_manager.cc:194] Registered new tserver with Master: ca994b55b43f415fae40a06233198f25 (127.9.117.194:36997)
I20251023 20:17:01.186828 30199 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.194:60599
W20251023 20:17:01.268868 30522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:01.269047 30522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:01.269073 30522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:01.270552 30522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:01.270601 30522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:01.270627 30522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.117.195
I20251023 20:17:01.272171 30522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.117.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.117.195
--webserver_port=0
--tserver_master_addrs=127.9.117.254:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.117.195
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:01.272445 30522 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:01.272702 30522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:01.275292 30530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:01.275338 30527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:01.275286 30528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:01.275509 30522 server_base.cc:1047] running on GCE node
I20251023 20:17:01.275646 30522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:01.275815 30522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:01.276942 30522 hybrid_clock.cc:648] HybridClock initialized: now 1761250621276931 us; error 27 us; skew 500 ppm
I20251023 20:17:01.278105 30522 webserver.cc:492] Webserver started at http://127.9.117.195:34273/ using document root <none> and password file <none>
I20251023 20:17:01.278280 30522 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:01.278317 30522 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:01.278415 30522 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:01.279369 30522 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/data/instance:
uuid: "0bef4b8a22f24af7ba235c1d40971f4d"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "a699f86d3a7a361b51c491e2228f6aef"
server_key_iv: "9680058ac7594a9bff9b9b93fbe85580"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.279660 30522 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance:
uuid: "0bef4b8a22f24af7ba235c1d40971f4d"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "a699f86d3a7a361b51c491e2228f6aef"
server_key_iv: "9680058ac7594a9bff9b9b93fbe85580"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.280925 30522 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:17:01.281669 30536 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.281879 30522 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:01.281955 30522 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/wal
uuid: "0bef4b8a22f24af7ba235c1d40971f4d"
format_stamp: "Formatted at 2025-10-23 20:17:01 on dist-test-slave-x62p"
server_key: "a699f86d3a7a361b51c491e2228f6aef"
server_key_iv: "9680058ac7594a9bff9b9b93fbe85580"
server_key_version: "encryptionkey@0"
I20251023 20:17:01.282027 30522 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:01.297899 30522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:01.298264 30522 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:01.298413 30522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:01.298672 30522 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:17:01.299033 30522 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:17:01.299073 30522 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.299113 30522 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:17:01.299147 30522 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:01.306016 30522 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.195:45921
I20251023 20:17:01.306071 30649 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.195:45921 every 8 connection(s)
I20251023 20:17:01.306392 30522 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/data/info.pb
I20251023 20:17:01.309693 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 30522
I20251023 20:17:01.309846 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/wal/instance
I20251023 20:17:01.310061 9687 external_mini_cluster.cc:1468] Setting key 8cb3d24710501c317beebbc808a540c5
I20251023 20:17:01.310912 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.307146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:54577 (local address 127.9.117.254:35135)
1023 20:17:01.307236 (+ 90us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.307239 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.307457 (+ 218us) server_negotiation.cc:408] Connection header received
1023 20:17:01.307980 (+ 523us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.307981 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.307998 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.308066 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:01.308558 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.309086 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.309727 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.309941 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.310586 (+ 645us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:01.310589 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:01.310589 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:01.310595 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:01.310614 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:01.310619 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:01.310674 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.310796 (+ 122us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.310831 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251023 20:17:01.311071 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.307109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:35135 (local address 127.9.117.195:54577)
1023 20:17:01.307342 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.307375 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:01.307895 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:01.308127 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:01.308130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:01.308139 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:01.308468 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.308470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.309167 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.309167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:01.309618 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.309620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.309739 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.310447 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1023 20:17:01.310448 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:01.310473 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1023 20:17:01.310494 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:01.310709 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:01.310715 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:01.310718 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:17:01.310789 (+ 71us) client_negotiation.cc:240] Negotiation successful
1023 20:17:01.310876 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20251023 20:17:01.311501 30650 heartbeater.cc:344] Connected to a master server at 127.9.117.254:35135
I20251023 20:17:01.311590 30650 heartbeater.cc:461] Registering TS with master...
I20251023 20:17:01.311800 30650 heartbeater.cc:507] Master 127.9.117.254:35135 requested a full tablet report, sending...
I20251023 20:17:01.312120 30199 ts_manager.cc:194] Registered new tserver with Master: 0bef4b8a22f24af7ba235c1d40971f4d (127.9.117.195:45921)
I20251023 20:17:01.312461 30199 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.117.195:54577
I20251023 20:17:01.322095 9687 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:17:01.325976 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.323462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.9.117.254:35135)
1023 20:17:01.323551 (+ 89us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.323554 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.323627 (+ 73us) server_negotiation.cc:408] Connection header received
1023 20:17:01.323686 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.323686 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.323695 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.323736 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:01.324050 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.324540 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.325146 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.325264 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.325757 (+ 493us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:01.325760 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:01.325761 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:01.325765 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:01.325776 (+ 11us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:01.325779 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:01.325825 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.325892 (+ 67us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.325914 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251023 20:17:01.327359 30199 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:17:01.328655 30199 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:17:01.334086 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.330659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:45957 (local address 127.0.0.1:35188)
1023 20:17:01.331629 (+ 970us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.331638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:01.331694 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:01.331860 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:01.331861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:01.331864 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:01.332063 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.332064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.333027 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.333028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:01.333785 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.333790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.333870 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.333916 (+ 46us) client_negotiation.cc:769] Sending connection context
1023 20:17:01.333964 (+ 48us) client_negotiation.cc:240] Negotiation successful
1023 20:17:01.334006 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":78,"threads_started":1}
I20251023 20:17:01.334391 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.330621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:36997 (local address 127.0.0.1:46842)
1023 20:17:01.330849 (+ 228us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.330880 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:01.330958 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:01.331252 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:01.331255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:01.331258 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:01.331520 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.331523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.333385 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.333386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:01.334140 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.334143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.334211 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.334215 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:17:01.334251 (+ 36us) client_negotiation.cc:240] Negotiation successful
1023 20:17:01.334296 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20251023 20:17:01.334903 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.330636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.195:45921 (local address 127.0.0.1:40976)
1023 20:17:01.331467 (+ 831us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.331477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:01.331539 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:01.332020 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:01.332021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:01.332023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:01.332205 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.332206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.333826 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.333827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:01.334692 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.334694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.334771 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.334774 (+ 3us) client_negotiation.cc:769] Sending connection context
1023 20:17:01.334804 (+ 30us) client_negotiation.cc:240] Negotiation successful
1023 20:17:01.334840 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":97,"threads_started":1}
I20251023 20:17:01.335003 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.331280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.9.117.193:45957)
1023 20:17:01.331538 (+ 258us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.331547 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.331698 (+ 151us) server_negotiation.cc:408] Connection header received
1023 20:17:01.331766 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.331769 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.331783 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.331855 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:01.332154 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.332956 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.334204 (+ 1248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.334725 (+ 521us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.334841 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.334871 (+ 30us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.334915 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":104,"threads_started":1}
I20251023 20:17:01.335325 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.330913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.9.117.194:36997)
1023 20:17:01.331074 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.331081 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.331091 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:17:01.331124 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.331126 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.331140 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.331202 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:01.331628 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.333283 (+ 1655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.334546 (+ 1263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.335085 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.335178 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.335208 (+ 30us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.335248 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20251023 20:17:01.335580 30321 tablet_service.cc:1505] Processing CreateTablet for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b (DEFAULT_TABLE table=test-table [id=ebd3c59374e8491c89a8153c849700d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:17:01.335880 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.330711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.9.117.195:45921)
1023 20:17:01.331842 (+ 1131us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.331851 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.331861 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:17:01.331909 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.331911 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.331931 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.332019 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:01.332284 (+ 265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.333744 (+ 1460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.334788 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.335670 (+ 882us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.335731 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.335762 (+ 31us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.335804 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":96,"threads_started":1}
I20251023 20:17:01.335944 30321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:01.335918 30452 tablet_service.cc:1505] Processing CreateTablet for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b (DEFAULT_TABLE table=test-table [id=ebd3c59374e8491c89a8153c849700d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:17:01.336187 30452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:01.336371 30584 tablet_service.cc:1505] Processing CreateTablet for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b (DEFAULT_TABLE table=test-table [id=ebd3c59374e8491c89a8153c849700d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:17:01.336633 30584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:01.338879 30664 tablet_bootstrap.cc:492] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: Bootstrap starting.
I20251023 20:17:01.339309 30665 tablet_bootstrap.cc:492] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: Bootstrap starting.
I20251023 20:17:01.339634 30664 tablet_bootstrap.cc:654] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:01.339866 30665 tablet_bootstrap.cc:654] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:01.339919 30664 log.cc:826] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:01.340142 30665 log.cc:826] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:01.340469 30666 tablet_bootstrap.cc:492] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156: Bootstrap starting.
I20251023 20:17:01.340612 30664 tablet_bootstrap.cc:492] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: No bootstrap required, opened a new log
I20251023 20:17:01.340680 30664 ts_tablet_manager.cc:1403] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:17:01.341048 30665 tablet_bootstrap.cc:492] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: No bootstrap required, opened a new log
I20251023 20:17:01.341127 30665 ts_tablet_manager.cc:1403] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:17:01.341156 30666 tablet_bootstrap.cc:654] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:01.341451 30666 log.cc:826] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:01.342222 30666 tablet_bootstrap.cc:492] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156: No bootstrap required, opened a new log
I20251023 20:17:01.342197 30664 raft_consensus.cc:359] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.342339 30666 ts_tablet_manager.cc:1403] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:17:01.342355 30664 raft_consensus.cc:385] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:01.342391 30664 raft_consensus.cc:740] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bef4b8a22f24af7ba235c1d40971f4d, State: Initialized, Role: FOLLOWER
I20251023 20:17:01.342527 30664 consensus_queue.cc:260] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.342805 30664 ts_tablet_manager.cc:1434] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:17:01.342772 30665 raft_consensus.cc:359] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.342904 30665 raft_consensus.cc:385] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:01.342842 30650 heartbeater.cc:499] Master 127.9.117.254:35135 was elected leader, sending a full tablet report...
I20251023 20:17:01.342933 30665 raft_consensus.cc:740] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca994b55b43f415fae40a06233198f25, State: Initialized, Role: FOLLOWER
I20251023 20:17:01.343024 30665 consensus_queue.cc:260] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.343267 30665 ts_tablet_manager.cc:1434] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:17:01.343329 30519 heartbeater.cc:499] Master 127.9.117.254:35135 was elected leader, sending a full tablet report...
I20251023 20:17:01.344033 30666 raft_consensus.cc:359] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.344188 30666 raft_consensus.cc:385] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:01.344226 30666 raft_consensus.cc:740] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1ad0e9eb694acf8ed00a8ed73f3156, State: Initialized, Role: FOLLOWER
I20251023 20:17:01.344312 30666 consensus_queue.cc:260] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.344506 30666 ts_tablet_manager.cc:1434] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:17:01.344573 30387 heartbeater.cc:499] Master 127.9.117.254:35135 was elected leader, sending a full tablet report...
I20251023 20:17:01.358786 30670 raft_consensus.cc:493] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:17:01.358950 30670 raft_consensus.cc:515] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.359424 30670 leader_election.cc:290] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca994b55b43f415fae40a06233198f25 (127.9.117.194:36997), 0e1ad0e9eb694acf8ed00a8ed73f3156 (127.9.117.193:45957)
I20251023 20:17:01.362219 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.359618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.194:36997 (local address 127.9.117.195:44175)
1023 20:17:01.359683 (+ 65us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.359695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:01.359749 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:01.359932 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:01.359932 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:01.359934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:01.360174 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.360177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.361117 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.361118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:01.361953 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.361956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.362039 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.362045 (+ 6us) client_negotiation.cc:769] Sending connection context
1023 20:17:01.362080 (+ 35us) client_negotiation.cc:240] Negotiation successful
1023 20:17:01.362137 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251023 20:17:01.362635 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.359618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:45957 (local address 127.9.117.195:41103)
1023 20:17:01.359869 (+ 251us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.359877 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:01.359935 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:01.360078 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:01.360079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:01.360080 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1023 20:17:01.360528 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.360531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.361463 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.361464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:01.362388 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:01.362391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.362455 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.362459 (+ 4us) client_negotiation.cc:769] Sending connection context
1023 20:17:01.362491 (+ 32us) client_negotiation.cc:240] Negotiation successful
1023 20:17:01.362531 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":14,"thread_start_us":144,"threads_started":1}
I20251023 20:17:01.362739 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.359707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:44175 (local address 127.9.117.194:36997)
1023 20:17:01.359762 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.359764 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.359776 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:17:01.359811 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.359812 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.359829 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.359905 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:01.360289 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.361024 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.362048 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.362579 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.362590 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.362617 (+ 27us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.362667 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:17:01.363113 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.359789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.117.195:41103 (local address 127.9.117.193:45957)
1023 20:17:01.359835 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.359838 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.359938 (+ 100us) server_negotiation.cc:408] Connection header received
1023 20:17:01.359990 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.359991 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.360002 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.360047 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1023 20:17:01.360632 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.361351 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.362518 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.363006 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.363015 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.363036 (+ 21us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.363064 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:17:01.363296 30472 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d2d0da9f2ef4f75bb6a5fafe25afb9b" candidate_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca994b55b43f415fae40a06233198f25" is_pre_election: true
I20251023 20:17:01.363484 30472 raft_consensus.cc:2468] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bef4b8a22f24af7ba235c1d40971f4d in term 0.
I20251023 20:17:01.363607 30341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d2d0da9f2ef4f75bb6a5fafe25afb9b" candidate_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" is_pre_election: true
I20251023 20:17:01.363760 30341 raft_consensus.cc:2468] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bef4b8a22f24af7ba235c1d40971f4d in term 0.
I20251023 20:17:01.363752 30537 leader_election.cc:304] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bef4b8a22f24af7ba235c1d40971f4d, ca994b55b43f415fae40a06233198f25; no voters:
I20251023 20:17:01.363905 30670 raft_consensus.cc:2804] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:17:01.363976 30670 raft_consensus.cc:493] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:17:01.364015 30670 raft_consensus.cc:3060] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:01.364776 30670 raft_consensus.cc:515] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.364943 30670 leader_election.cc:290] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [CANDIDATE]: Term 1 election: Requested vote from peers ca994b55b43f415fae40a06233198f25 (127.9.117.194:36997), 0e1ad0e9eb694acf8ed00a8ed73f3156 (127.9.117.193:45957)
I20251023 20:17:01.365113 30472 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d2d0da9f2ef4f75bb6a5fafe25afb9b" candidate_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca994b55b43f415fae40a06233198f25"
I20251023 20:17:01.365170 30341 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d2d0da9f2ef4f75bb6a5fafe25afb9b" candidate_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156"
I20251023 20:17:01.365211 30472 raft_consensus.cc:3060] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:01.365293 30341 raft_consensus.cc:3060] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:01.365988 30472 raft_consensus.cc:2468] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bef4b8a22f24af7ba235c1d40971f4d in term 1.
I20251023 20:17:01.366080 30341 raft_consensus.cc:2468] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bef4b8a22f24af7ba235c1d40971f4d in term 1.
I20251023 20:17:01.366216 30537 leader_election.cc:304] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bef4b8a22f24af7ba235c1d40971f4d, ca994b55b43f415fae40a06233198f25; no voters:
I20251023 20:17:01.366343 30670 raft_consensus.cc:2804] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:01.366528 30670 raft_consensus.cc:697] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 1 LEADER]: Becoming Leader. State: Replica: 0bef4b8a22f24af7ba235c1d40971f4d, State: Running, Role: LEADER
I20251023 20:17:01.366655 30670 consensus_queue.cc:237] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } }
I20251023 20:17:01.367568 30199 catalog_manager.cc:5649] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bef4b8a22f24af7ba235c1d40971f4d (127.9.117.195). New cstate: current_term: 1 leader_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bef4b8a22f24af7ba235c1d40971f4d" member_type: VOTER last_known_addr { host: "127.9.117.195" port: 45921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 } health_report { overall_health: UNKNOWN } } }
I20251023 20:17:01.374711 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.372481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.9.117.195:45921)
1023 20:17:01.372543 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:01.372546 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:01.372557 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:17:01.372606 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:01.372606 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:01.372630 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:01.372690 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:01.373037 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.373604 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:01.374211 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:01.374334 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:01.374381 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:01.374496 (+ 115us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:01.374560 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:01.374613 (+ 53us) server_negotiation.cc:300] Negotiation successful
1023 20:17:01.374641 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251023 20:17:01.376689 30472 raft_consensus.cc:1275] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 1 FOLLOWER]: Refusing update from remote peer 0bef4b8a22f24af7ba235c1d40971f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:17:01.376685 30341 raft_consensus.cc:1275] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0e1ad0e9eb694acf8ed00a8ed73f3156 [term 1 FOLLOWER]: Refusing update from remote peer 0bef4b8a22f24af7ba235c1d40971f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:17:01.377113 30674 consensus_queue.cc:1048] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1ad0e9eb694acf8ed00a8ed73f3156" member_type: VOTER last_known_addr { host: "127.9.117.193" port: 45957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:17:01.377251 30670 consensus_queue.cc:1048] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca994b55b43f415fae40a06233198f25" member_type: VOTER last_known_addr { host: "127.9.117.194" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:17:01.380561 30677 mvcc.cc:204] Tried to move back new op lower bound from 7214082545155981312 to 7214082545118666752. Current Snapshot: MvccSnapshot[applied={T|T < 7214082545155981312 or (T in {7214082545155981312})}]
I20251023 20:17:01.383060 30198 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42224:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:17:01.383183 30198 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42224:
table { table_name: "test-table" } modify_external_catalogs: true
I20251023 20:17:01.384037 30198 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 78976784376647f79d43d71bc212ddd3: Sending DeleteTablet for 3 replicas of tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b
I20251023 20:17:01.384392 30452 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:17:01 UTC) from {username='slave'} at 127.0.0.1:46842
I20251023 20:17:01.384423 30584 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:17:01 UTC) from {username='slave'} at 127.0.0.1:40976
I20251023 20:17:01.384763 30687 tablet_replica.cc:333] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: stopping tablet replica
I20251023 20:17:01.384752 30321 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:17:01 UTC) from {username='slave'} at 127.0.0.1:35188
I20251023 20:17:01.384845 30688 tablet_replica.cc:333] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: stopping tablet replica
I20251023 20:17:01.384891 30687 raft_consensus.cc:2243] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:17:01.384919 30688 raft_consensus.cc:2243] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:17:01.385006 30687 raft_consensus.cc:2272] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:17:01.385043 30688 raft_consensus.cc:2272] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:17:01.385358 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 30259
I20251023 20:17:01.385788 30688 ts_tablet_manager.cc:1916] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:17:01.385792 30687 ts_tablet_manager.cc:1916] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:17:01.387490 30687 ts_tablet_manager.cc:1929] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:17:01.387490 30688 ts_tablet_manager.cc:1929] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:17:01.387560 30688 log.cc:1199] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-2/wal/wals/1d2d0da9f2ef4f75bb6a5fafe25afb9b
I20251023 20:17:01.387558 30687 log.cc:1199] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: Deleting WAL directory at /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761250300132822-9687-0/minicluster-data/ts-1/wal/wals/1d2d0da9f2ef4f75bb6a5fafe25afb9b
I20251023 20:17:01.387846 30688 ts_tablet_manager.cc:1950] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P 0bef4b8a22f24af7ba235c1d40971f4d: Deleting consensus metadata
I20251023 20:17:01.387846 30687 ts_tablet_manager.cc:1950] T 1d2d0da9f2ef4f75bb6a5fafe25afb9b P ca994b55b43f415fae40a06233198f25: Deleting consensus metadata
I20251023 20:17:01.388244 30185 catalog_manager.cc:4985] TS 0bef4b8a22f24af7ba235c1d40971f4d (127.9.117.195:45921): tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b (table test-table [id=ebd3c59374e8491c89a8153c849700d4]) successfully deleted
I20251023 20:17:01.389370 30183 catalog_manager.cc:4985] TS ca994b55b43f415fae40a06233198f25 (127.9.117.194:36997): tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b (table test-table [id=ebd3c59374e8491c89a8153c849700d4]) successfully deleted
W20251023 20:17:01.392779 30186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.117.193:45957 (error 108)
I20251023 20:17:01.393390 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:01.393242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.193:45957 (local address 127.0.0.1:35200)
1023 20:17:01.393309 (+ 67us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:01.393355 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.193:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251023 20:17:01.393424 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 30390
W20251023 20:17:01.396165 30186 catalog_manager.cc:4712] TS 0e1ad0e9eb694acf8ed00a8ed73f3156 (127.9.117.193:45957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d2d0da9f2ef4f75bb6a5fafe25afb9b: Network error: Client connection negotiation failed: client connection to 127.9.117.193:45957: connect: Connection refused (error 111)
I20251023 20:17:01.401293 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 30522
I20251023 20:17:01.406050 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 30168
2025-10-23T20:17:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (656 ms)
[----------] 6 tests from EncryptionPolicyTest (11745 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:01 dist-test-slave-x62p krb5kdc[30693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:01 dist-test-slave-x62p krb5kdc[30693](info): set up 2 sockets
Oct 23 20:17:01 dist-test-slave-x62p krb5kdc[30693](info): commencing operation
krb5kdc: starting...
W20251023 20:17:03.429454 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.003s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:03 dist-test-slave-x62p krb5kdc[30693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:03.569622 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:03.656323 30709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:03.656630 30709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:03.656683 30709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:03.658432 30709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:03.658497 30709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:03.658524 30709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:03.658551 30709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:03.658572 30709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:03.660507 30709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:03.660844 30709 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:03.661046 30709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:03.663992 30714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:03.664067 30717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:03.663993 30715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:03.664146 30709 server_base.cc:1047] running on GCE node
I20251023 20:17:03.664547 30709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:03.664873 30709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:03.666018 30709 hybrid_clock.cc:648] HybridClock initialized: now 1761250623666004 us; error 33 us; skew 500 ppm
Oct 23 20:17:03 dist-test-slave-x62p krb5kdc[30693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:03.668766 30709 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:03.669409 30709 webserver.cc:492] Webserver started at http://127.0.0.1:36023/ using document root <none> and password file <none>
I20251023 20:17:03.669643 30709 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:03.669689 30709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:03.669821 30709 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:03.670871 30709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "ed3e555e60684f52925e394a5f96bde9"
format_stamp: "Formatted at 2025-10-23 20:17:03 on dist-test-slave-x62p"
server_key: "d1ba66b060a909a90900259e91f23f11"
server_key_iv: "43cd5c080dcdfc8857772fd51d5f381c"
server_key_version: "encryptionkey@0"
I20251023 20:17:03.671216 30709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "ed3e555e60684f52925e394a5f96bde9"
format_stamp: "Formatted at 2025-10-23 20:17:03 on dist-test-slave-x62p"
server_key: "d1ba66b060a909a90900259e91f23f11"
server_key_iv: "43cd5c080dcdfc8857772fd51d5f381c"
server_key_version: "encryptionkey@0"
I20251023 20:17:03.672524 30709 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:17:03.673301 30724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:03.673513 30709 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:03.673614 30709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "ed3e555e60684f52925e394a5f96bde9"
format_stamp: "Formatted at 2025-10-23 20:17:03 on dist-test-slave-x62p"
server_key: "d1ba66b060a909a90900259e91f23f11"
server_key_iv: "43cd5c080dcdfc8857772fd51d5f381c"
server_key_version: "encryptionkey@0"
I20251023 20:17:03.673679 30709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:03.691203 30709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:03.693449 30709 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:03.693614 30709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:03.697971 30709 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45057
I20251023 20:17:03.698017 30776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:45057 every 8 connection(s)
I20251023 20:17:03.698364 30709 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:03.699014 30777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:03.701905 30777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9: Bootstrap starting.
I20251023 20:17:03.702564 30777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:03.702831 30777 log.cc:826] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:03.703593 30777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9: No bootstrap required, opened a new log
I20251023 20:17:03.704838 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 30709
I20251023 20:17:03.704972 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:03.705168 9687 external_mini_cluster.cc:1468] Setting key fb904c9a4a832383232a0fb4bbd8153b
I20251023 20:17:03.705168 30777 raft_consensus.cc:359] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e555e60684f52925e394a5f96bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20251023 20:17:03.705335 30777 raft_consensus.cc:385] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:03.705385 30777 raft_consensus.cc:740] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3e555e60684f52925e394a5f96bde9, State: Initialized, Role: FOLLOWER
I20251023 20:17:03.705534 30777 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e555e60684f52925e394a5f96bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20251023 20:17:03.705610 30777 raft_consensus.cc:399] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:03.705683 30777 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:03.705739 30777 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:03.706339 30777 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e555e60684f52925e394a5f96bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20251023 20:17:03.706454 30777 leader_election.cc:304] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed3e555e60684f52925e394a5f96bde9; no voters:
I20251023 20:17:03.706606 30777 leader_election.cc:290] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:03.706658 30782 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:03.706825 30782 raft_consensus.cc:697] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [term 1 LEADER]: Becoming Leader. State: Replica: ed3e555e60684f52925e394a5f96bde9, State: Running, Role: LEADER
I20251023 20:17:03.706856 30777 sys_catalog.cc:565] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:03.707012 30782 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e555e60684f52925e394a5f96bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20251023 20:17:03.708695 30784 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed3e555e60684f52925e394a5f96bde9. Latest consensus state: current_term: 1 leader_uuid: "ed3e555e60684f52925e394a5f96bde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e555e60684f52925e394a5f96bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } } }
I20251023 20:17:03.708711 30783 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed3e555e60684f52925e394a5f96bde9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3e555e60684f52925e394a5f96bde9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } } }
I20251023 20:17:03.708842 30784 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:03.708849 30783 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9 [sys.catalog]: This master's current role is: LEADER
W20251023 20:17:03.708967 30796 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:03.708997 30796 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:03.709148 30798 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:03.709343 30798 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 23 20:17:03 dist-test-slave-x62p krb5kdc[30693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251023 20:17:03.715431 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:03.705711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60709 (local address 127.0.0.1:45057)
1023 20:17:03.705926 (+ 215us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:03.705940 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:03.705972 (+ 32us) server_negotiation.cc:408] Connection header received
1023 20:17:03.706093 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:03.706098 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:03.706151 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:03.706277 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:03.707280 (+ 1003us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:03.708165 (+ 885us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:03.709075 (+ 910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:03.709352 (+ 277us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:03.711951 (+ 2599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:03.711962 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:03.711962 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:03.711982 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:03.713935 (+ 1953us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:03.714404 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:03.714407 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:03.714407 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:03.714458 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:03.714643 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:03.714643 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:03.714644 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:03.714844 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:03.714957 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:03.715123 (+ 166us) server_negotiation.cc:300] Negotiation successful
1023 20:17:03.715206 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20251023 20:17:03.716583 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:03.716871 30798 catalog_manager.cc:1357] Generated new cluster ID: f70a5cd45851410394fd6c34ec103416
I20251023 20:17:03.716921 30798 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:03.720695 30798 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:03.721552 30798 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:03.725585 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:03.718285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45851 (local address 127.0.0.1:45057)
1023 20:17:03.718334 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:03.718336 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:03.718441 (+ 105us) server_negotiation.cc:408] Connection header received
1023 20:17:03.718495 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:03.718497 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:03.718507 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:03.718578 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:03.719395 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:03.720216 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:03.720895 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:03.721081 (+ 186us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:03.722692 (+ 1611us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:03.722697 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:03.722697 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:03.722705 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:03.724451 (+ 1746us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:03.724887 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:03.724888 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:03.724889 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:03.724914 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:03.725097 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:03.725098 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:03.725098 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:03.725205 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:03.725309 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:03.725427 (+ 118us) server_negotiation.cc:300] Negotiation successful
1023 20:17:03.725494 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:17:03.727252 30798 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ed3e555e60684f52925e394a5f96bde9: Generated new TSK 0
I20251023 20:17:03.727432 30798 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:03.732791 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 30709
W20251023 20:17:03.737537 30691 connection.cc:537] client connection to 127.0.0.1:45057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45057): Connection reset by peer (error 104)
2025-10-23T20:17:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:03 dist-test-slave-x62p krb5kdc[30808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:03 dist-test-slave-x62p krb5kdc[30808](info): set up 2 sockets
Oct 23 20:17:03 dist-test-slave-x62p krb5kdc[30808](info): commencing operation
krb5kdc: starting...
W20251023 20:17:05.729182 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:05 dist-test-slave-x62p krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:05.880179 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:05.964661 30824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:05.964921 30824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:05.964967 30824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:05.966508 30824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:05.966562 30824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:05.966586 30824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:05.966609 30824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:05.966629 30824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:05.968333 30824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:05.968604 30824 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:05.968941 30824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:05.971702 30830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:05.971702 30829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:05.971801 30824 server_base.cc:1047] running on GCE node
W20251023 20:17:05.971702 30832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:05.972124 30824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:05.972383 30824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:05.973558 30824 hybrid_clock.cc:648] HybridClock initialized: now 1761250625973521 us; error 48 us; skew 500 ppm
Oct 23 20:17:05 dist-test-slave-x62p krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:05.976176 30824 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:05.976796 30824 webserver.cc:492] Webserver started at http://127.0.0.1:46673/ using document root <none> and password file <none>
I20251023 20:17:05.977006 30824 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:05.977051 30824 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:05.977174 30824 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:05.978214 30824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "816602dc173c432c9130c2fb7ebedb5f"
format_stamp: "Formatted at 2025-10-23 20:17:05 on dist-test-slave-x62p"
server_key: "4913e1f1cb505629c45d366dd61213a3"
server_key_iv: "512ee292ffc4fd715b415125ce380658"
server_key_version: "encryptionkey@0"
I20251023 20:17:05.978544 30824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "816602dc173c432c9130c2fb7ebedb5f"
format_stamp: "Formatted at 2025-10-23 20:17:05 on dist-test-slave-x62p"
server_key: "4913e1f1cb505629c45d366dd61213a3"
server_key_iv: "512ee292ffc4fd715b415125ce380658"
server_key_version: "encryptionkey@0"
I20251023 20:17:05.979893 30824 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:17:05.980628 30839 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:05.980906 30824 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:05.980998 30824 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "816602dc173c432c9130c2fb7ebedb5f"
format_stamp: "Formatted at 2025-10-23 20:17:05 on dist-test-slave-x62p"
server_key: "4913e1f1cb505629c45d366dd61213a3"
server_key_iv: "512ee292ffc4fd715b415125ce380658"
server_key_version: "encryptionkey@0"
I20251023 20:17:05.981074 30824 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:06.002599 30824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:06.004877 30824 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:06.005043 30824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:06.009306 30824 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37459
I20251023 20:17:06.009383 30891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:37459 every 8 connection(s)
I20251023 20:17:06.009753 30824 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:06.010417 30892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:06.013181 30892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f: Bootstrap starting.
I20251023 20:17:06.013787 30892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:06.014020 30892 log.cc:826] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:06.014739 30892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f: No bootstrap required, opened a new log
I20251023 20:17:06.015514 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 30824
I20251023 20:17:06.015672 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:06.015864 9687 external_mini_cluster.cc:1468] Setting key 6339cbdbe17a7c03ee771c47fc383989
I20251023 20:17:06.016463 30892 raft_consensus.cc:359] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816602dc173c432c9130c2fb7ebedb5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20251023 20:17:06.016640 30892 raft_consensus.cc:385] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:06.016669 30892 raft_consensus.cc:740] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816602dc173c432c9130c2fb7ebedb5f, State: Initialized, Role: FOLLOWER
I20251023 20:17:06.016812 30892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816602dc173c432c9130c2fb7ebedb5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20251023 20:17:06.016878 30892 raft_consensus.cc:399] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:06.016901 30892 raft_consensus.cc:493] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:06.016954 30892 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:06.017578 30892 raft_consensus.cc:515] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816602dc173c432c9130c2fb7ebedb5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20251023 20:17:06.017683 30892 leader_election.cc:304] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816602dc173c432c9130c2fb7ebedb5f; no voters:
I20251023 20:17:06.017865 30892 leader_election.cc:290] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:06.017920 30897 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:06.018036 30897 raft_consensus.cc:697] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [term 1 LEADER]: Becoming Leader. State: Replica: 816602dc173c432c9130c2fb7ebedb5f, State: Running, Role: LEADER
I20251023 20:17:06.018143 30892 sys_catalog.cc:565] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:06.018193 30897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816602dc173c432c9130c2fb7ebedb5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20251023 20:17:06.018610 30899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "816602dc173c432c9130c2fb7ebedb5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816602dc173c432c9130c2fb7ebedb5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } } }
I20251023 20:17:06.018692 30899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:06.018831 30899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816602dc173c432c9130c2fb7ebedb5f. Latest consensus state: current_term: 1 leader_uuid: "816602dc173c432c9130c2fb7ebedb5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816602dc173c432c9130c2fb7ebedb5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } } }
I20251023 20:17:06.018888 30899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:06.019367 30902 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:06.019937 30902 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:06.021456 30896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:06.016427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46389 (local address 127.0.0.1:37459)
1023 20:17:06.016641 (+ 214us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:06.016655 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:06.016684 (+ 29us) server_negotiation.cc:408] Connection header received
1023 20:17:06.016849 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:06.016855 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:06.016913 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:06.017072 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:06.017487 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:06.018632 (+ 1145us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:06.019730 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:06.020349 (+ 619us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:06.020687 (+ 338us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:06.020694 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:06.020695 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:06.020947 (+ 252us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:06.020970 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:06.020979 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:06.021082 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:06.021244 (+ 162us) server_negotiation.cc:300] Negotiation successful
1023 20:17:06.021276 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20251023 20:17:06.021763 30902 catalog_manager.cc:1357] Generated new cluster ID: b25d7c0c906640a0aa73b61be4c7007b
I20251023 20:17:06.021816 30902 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:06.022436 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:06.027416 30896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:06.023867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39761 (local address 127.0.0.1:37459)
1023 20:17:06.024847 (+ 980us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:06.024849 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:06.024857 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:17:06.024879 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:06.024879 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:06.024890 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:06.024947 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:06.025250 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:06.025728 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:06.026286 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:06.026513 (+ 227us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:06.027099 (+ 586us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:06.027104 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:06.027104 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:06.027110 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:06.027130 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:06.027135 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:06.027206 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:06.027287 (+ 81us) server_negotiation.cc:300] Negotiation successful
1023 20:17:06.027301 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951}
I20251023 20:17:06.041405 30902 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:06.042019 30902 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:06.047334 30902 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 816602dc173c432c9130c2fb7ebedb5f: Generated new TSK 0
I20251023 20:17:06.047569 30902 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:06.051406 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 30824
W20251023 20:17:06.056034 30806 connection.cc:537] client connection to 127.0.0.1:37459 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37459): Connection reset by peer (error 104)
2025-10-23T20:17:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:06 dist-test-slave-x62p krb5kdc[30923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:06 dist-test-slave-x62p krb5kdc[30923](info): set up 2 sockets
Oct 23 20:17:06 dist-test-slave-x62p krb5kdc[30923](info): commencing operation
krb5kdc: starting...
W20251023 20:17:08.081418 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:08 dist-test-slave-x62p krb5kdc[30923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:08.219923 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:08.305091 30939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:08.305301 30939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:08.305327 30939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:08.306918 30939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:08.306978 30939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:08.306993 30939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:08.307006 30939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:08.307018 30939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:08.308827 30939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:08.309082 30939 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:08.309283 30939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:08.311931 30947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:08.312016 30945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:08.312013 30944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:08.312186 30939 server_base.cc:1047] running on GCE node
I20251023 20:17:08.312529 30939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:08.312966 30939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:08.314147 30939 hybrid_clock.cc:648] HybridClock initialized: now 1761250628314124 us; error 36 us; skew 500 ppm
Oct 23 20:17:08 dist-test-slave-x62p krb5kdc[30923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:08.316874 30939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:08.317533 30939 webserver.cc:492] Webserver started at http://127.0.0.1:38621/ using document root <none> and password file <none>
I20251023 20:17:08.317745 30939 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:08.317790 30939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:08.317903 30939 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:08.318912 30939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "e7c2de45c5e54a019bb11e32a09e2d9b"
format_stamp: "Formatted at 2025-10-23 20:17:08 on dist-test-slave-x62p"
server_key: "35b04dfe4e6ebe3a18c49d678197732f"
server_key_iv: "6ea00c64b3f75916583c6619ab2f684c"
server_key_version: "encryptionkey@0"
I20251023 20:17:08.319255 30939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "e7c2de45c5e54a019bb11e32a09e2d9b"
format_stamp: "Formatted at 2025-10-23 20:17:08 on dist-test-slave-x62p"
server_key: "35b04dfe4e6ebe3a18c49d678197732f"
server_key_iv: "6ea00c64b3f75916583c6619ab2f684c"
server_key_version: "encryptionkey@0"
I20251023 20:17:08.320539 30939 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:17:08.321379 30954 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:08.321601 30939 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:08.321693 30939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "e7c2de45c5e54a019bb11e32a09e2d9b"
format_stamp: "Formatted at 2025-10-23 20:17:08 on dist-test-slave-x62p"
server_key: "35b04dfe4e6ebe3a18c49d678197732f"
server_key_iv: "6ea00c64b3f75916583c6619ab2f684c"
server_key_version: "encryptionkey@0"
I20251023 20:17:08.321758 30939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:08.325176 30939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:08.327023 30939 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:08.327167 30939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:08.331369 30939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46553
I20251023 20:17:08.331398 31006 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:46553 every 8 connection(s)
I20251023 20:17:08.331740 30939 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:08.332327 31007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:08.334908 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 30939
I20251023 20:17:08.335047 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:08.335031 31007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b: Bootstrap starting.
I20251023 20:17:08.335232 9687 external_mini_cluster.cc:1468] Setting key 1f9a67d46444941032eeb74dabbd5905
I20251023 20:17:08.335817 31007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:08.336134 31007 log.cc:826] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:08.336998 31007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b: No bootstrap required, opened a new log
I20251023 20:17:08.337816 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:08.335857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45921 (local address 127.0.0.1:46553)
1023 20:17:08.336045 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:08.336057 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:08.336088 (+ 31us) server_negotiation.cc:408] Connection header received
1023 20:17:08.336498 (+ 410us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:08.336504 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:08.336532 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:08.336664 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:08.337378 (+ 714us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:08.337382 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:08.337382 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:08.337387 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:08.337418 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:08.337427 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:08.337521 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:08.337612 (+ 91us) server_negotiation.cc:300] Negotiation successful
1023 20:17:08.337649 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20251023 20:17:08.338413 31007 raft_consensus.cc:359] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c2de45c5e54a019bb11e32a09e2d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46553 } }
I20251023 20:17:08.338570 31007 raft_consensus.cc:385] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:08.338627 31007 raft_consensus.cc:740] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c2de45c5e54a019bb11e32a09e2d9b, State: Initialized, Role: FOLLOWER
I20251023 20:17:08.338722 31007 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c2de45c5e54a019bb11e32a09e2d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46553 } }
I20251023 20:17:08.338798 31007 raft_consensus.cc:399] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:08.338850 31007 raft_consensus.cc:493] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:08.338901 31007 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:08.338889 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:08.339483 31007 raft_consensus.cc:515] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c2de45c5e54a019bb11e32a09e2d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46553 } }
I20251023 20:17:08.339602 31007 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c2de45c5e54a019bb11e32a09e2d9b; no voters:
I20251023 20:17:08.339761 31007 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:08.339828 31012 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:08.340034 31012 raft_consensus.cc:697] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [term 1 LEADER]: Becoming Leader. State: Replica: e7c2de45c5e54a019bb11e32a09e2d9b, State: Running, Role: LEADER
I20251023 20:17:08.340083 31007 sys_catalog.cc:565] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:08.340200 31012 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c2de45c5e54a019bb11e32a09e2d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46553 } }
I20251023 20:17:08.340711 31012 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c2de45c5e54a019bb11e32a09e2d9b. Latest consensus state: current_term: 1 leader_uuid: "e7c2de45c5e54a019bb11e32a09e2d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c2de45c5e54a019bb11e32a09e2d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46553 } } }
I20251023 20:17:08.340900 31012 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:08.341151 31026 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:08.341253 31015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c2de45c5e54a019bb11e32a09e2d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c2de45c5e54a019bb11e32a09e2d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46553 } } }
I20251023 20:17:08.341360 31015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:08.341802 31026 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:08.341908 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:08.340352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45119 (local address 127.0.0.1:46553)
1023 20:17:08.340431 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:08.340433 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:08.340544 (+ 111us) server_negotiation.cc:408] Connection header received
1023 20:17:08.340594 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:08.340594 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:08.340601 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:08.340667 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:08.341607 (+ 940us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:08.341609 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:08.341609 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:08.341615 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:08.341632 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:08.341637 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:08.341672 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:08.341804 (+ 132us) server_negotiation.cc:300] Negotiation successful
1023 20:17:08.341822 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251023 20:17:08.343470 31026 catalog_manager.cc:1357] Generated new cluster ID: 64a7f760e1974448ab6547c597a58b33
I20251023 20:17:08.343525 31026 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:08.349584 31026 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:08.349776 31026 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:08.371142 31026 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e7c2de45c5e54a019bb11e32a09e2d9b: Generated new TSK 0
I20251023 20:17:08.371392 31026 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:08.373222 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 30939
2025-10-23T20:17:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:08 dist-test-slave-x62p krb5kdc[31038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:08 dist-test-slave-x62p krb5kdc[31038](info): set up 2 sockets
Oct 23 20:17:08 dist-test-slave-x62p krb5kdc[31038](info): commencing operation
krb5kdc: starting...
W20251023 20:17:10.385370 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.993s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:10 dist-test-slave-x62p krb5kdc[31038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:10.527343 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:10.610982 31054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:10.611194 31054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:10.611222 31054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:10.612779 31054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:10.612845 31054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:10.612861 31054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:10.612875 31054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:10.612887 31054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:10.614610 31054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:10.614827 31054 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:10.614979 31054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:10.617730 31059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:10.617735 31060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:10.617739 31062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:10.617975 31054 server_base.cc:1047] running on GCE node
I20251023 20:17:10.618248 31054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:10.618482 31054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:10.619652 31054 hybrid_clock.cc:648] HybridClock initialized: now 1761250630619630 us; error 32 us; skew 500 ppm
Oct 23 20:17:10 dist-test-slave-x62p krb5kdc[31038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:10.622331 31054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:10.622972 31054 webserver.cc:492] Webserver started at http://127.0.0.1:33739/ using document root <none> and password file <none>
I20251023 20:17:10.623175 31054 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:10.623217 31054 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:10.623339 31054 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:10.624328 31054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae"
format_stamp: "Formatted at 2025-10-23 20:17:10 on dist-test-slave-x62p"
server_key: "7e5617be6bba77de243730385a2a79a5"
server_key_iv: "7046427b300d36a3e2726e1c56c8c314"
server_key_version: "encryptionkey@0"
I20251023 20:17:10.624648 31054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae"
format_stamp: "Formatted at 2025-10-23 20:17:10 on dist-test-slave-x62p"
server_key: "7e5617be6bba77de243730385a2a79a5"
server_key_iv: "7046427b300d36a3e2726e1c56c8c314"
server_key_version: "encryptionkey@0"
I20251023 20:17:10.625998 31054 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:10.626822 31069 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:10.627063 31054 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:10.627146 31054 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae"
format_stamp: "Formatted at 2025-10-23 20:17:10 on dist-test-slave-x62p"
server_key: "7e5617be6bba77de243730385a2a79a5"
server_key_iv: "7046427b300d36a3e2726e1c56c8c314"
server_key_version: "encryptionkey@0"
I20251023 20:17:10.627216 31054 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:10.662990 31054 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:10.665227 31054 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:10.665374 31054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:10.669782 31054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44653
I20251023 20:17:10.669811 31122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:44653 every 8 connection(s)
I20251023 20:17:10.670159 31054 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:10.670840 31123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:10.672648 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31054
I20251023 20:17:10.672832 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:10.673017 9687 external_mini_cluster.cc:1468] Setting key 547c3d9441905df40e1d1a127000538f
I20251023 20:17:10.674129 31123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae: Bootstrap starting.
I20251023 20:17:10.674970 31123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:10.675295 31123 log.cc:826] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:10.676054 31123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae: No bootstrap required, opened a new log
I20251023 20:17:10.677663 31123 raft_consensus.cc:359] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44653 } }
I20251023 20:17:10.677832 31123 raft_consensus.cc:385] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:10.677865 31123 raft_consensus.cc:740] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781b15a8e5c74ecba7ea1f7fbefe98ae, State: Initialized, Role: FOLLOWER
I20251023 20:17:10.678017 31123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44653 } }
I20251023 20:17:10.678084 31123 raft_consensus.cc:399] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:10.678117 31123 raft_consensus.cc:493] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:10.678169 31123 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:10.678807 31123 raft_consensus.cc:515] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44653 } }
I20251023 20:17:10.678920 31123 leader_election.cc:304] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 781b15a8e5c74ecba7ea1f7fbefe98ae; no voters:
I20251023 20:17:10.679083 31123 leader_election.cc:290] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:10.679136 31128 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:10.679263 31128 raft_consensus.cc:697] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [term 1 LEADER]: Becoming Leader. State: Replica: 781b15a8e5c74ecba7ea1f7fbefe98ae, State: Running, Role: LEADER
I20251023 20:17:10.679397 31123 sys_catalog.cc:565] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:10.679411 31128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44653 } }
Oct 23 20:17:10 dist-test-slave-x62p krb5kdc[31038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251023 20:17:10.679881 31128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781b15a8e5c74ecba7ea1f7fbefe98ae. Latest consensus state: current_term: 1 leader_uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44653 } } }
I20251023 20:17:10.679970 31128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:10.680455 31135 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:10.680641 31129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b15a8e5c74ecba7ea1f7fbefe98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44653 } } }
I20251023 20:17:10.680874 31129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:10.682220 31135 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:10.684056 31135 catalog_manager.cc:1357] Generated new cluster ID: 5419d23d88a94ab0a67ea2f322e5d404
I20251023 20:17:10.684118 31135 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:10.685163 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:10.673598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46207 (local address 127.0.0.1:44653)
1023 20:17:10.673777 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:10.673788 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:10.673809 (+ 21us) server_negotiation.cc:408] Connection header received
1023 20:17:10.674128 (+ 319us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:10.674133 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:10.674177 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:10.674315 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:10.675393 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:10.676354 (+ 961us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:10.677306 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:10.677619 (+ 313us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:10.680479 (+ 2860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:10.680492 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:10.680493 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:10.680529 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:10.682940 (+ 2411us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:10.683420 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:10.683423 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:10.683424 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:10.683489 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:10.683691 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:10.683692 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:10.683693 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:10.683977 (+ 284us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:10.684138 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:10.684827 (+ 689us) server_negotiation.cc:300] Negotiation successful
1023 20:17:10.684976 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20251023 20:17:10.686169 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:10.698618 31135 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:10.699426 31135 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:10.700768 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:10.693161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36895 (local address 127.0.0.1:44653)
1023 20:17:10.693228 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:10.693231 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:10.693320 (+ 89us) server_negotiation.cc:408] Connection header received
1023 20:17:10.693405 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:10.693406 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:10.693418 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:10.693581 (+ 163us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:10.694466 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:10.695261 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:10.696023 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:10.696690 (+ 667us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:10.697843 (+ 1153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:10.697850 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:10.697850 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:10.697860 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:10.699662 (+ 1802us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:10.700056 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:10.700057 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:10.700058 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:10.700082 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:10.700218 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:10.700218 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:10.700219 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:10.700323 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:10.700428 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:10.700534 (+ 106us) server_negotiation.cc:300] Negotiation successful
1023 20:17:10.700638 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:17:10.706861 31135 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 781b15a8e5c74ecba7ea1f7fbefe98ae: Generated new TSK 0
I20251023 20:17:10.707082 31135 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:10.710248 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31054
2025-10-23T20:17:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:10 dist-test-slave-x62p krb5kdc[31154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:10 dist-test-slave-x62p krb5kdc[31154](info): set up 2 sockets
Oct 23 20:17:10 dist-test-slave-x62p krb5kdc[31154](info): commencing operation
krb5kdc: starting...
W20251023 20:17:12.717244 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.987s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:12 dist-test-slave-x62p krb5kdc[31154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:12.864897 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:12.947997 31170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:12.948189 31170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:12.948211 31170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:12.949718 31170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:12.949769 31170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:12.949790 31170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:12.949803 31170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:12.949817 31170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:12.951444 31170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:12.951673 31170 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:12.951874 31170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:12.954509 31178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:12.954504 31175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:12.954711 31170 server_base.cc:1047] running on GCE node
W20251023 20:17:12.954504 31176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:12.955111 31170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:12.955394 31170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:12.956568 31170 hybrid_clock.cc:648] HybridClock initialized: now 1761250632956545 us; error 33 us; skew 500 ppm
Oct 23 20:17:12 dist-test-slave-x62p krb5kdc[31154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:12.959064 31170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:12.959604 31170 webserver.cc:492] Webserver started at http://127.0.0.1:41963/ using document root <none> and password file <none>
I20251023 20:17:12.959774 31170 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:12.959806 31170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:12.959889 31170 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:12.960887 31170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "417d53cf81ad44d5b59c3ed750c252b5"
format_stamp: "Formatted at 2025-10-23 20:17:12 on dist-test-slave-x62p"
server_key: "cfeb7f77208c96e554fd48d036cec2b0"
server_key_iv: "f28a57325f7e81de8e67d4e8f9924eec"
server_key_version: "encryptionkey@0"
I20251023 20:17:12.961162 31170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "417d53cf81ad44d5b59c3ed750c252b5"
format_stamp: "Formatted at 2025-10-23 20:17:12 on dist-test-slave-x62p"
server_key: "cfeb7f77208c96e554fd48d036cec2b0"
server_key_iv: "f28a57325f7e81de8e67d4e8f9924eec"
server_key_version: "encryptionkey@0"
I20251023 20:17:12.962345 31170 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:12.963047 31185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:12.963235 31170 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:17:12.963305 31170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "417d53cf81ad44d5b59c3ed750c252b5"
format_stamp: "Formatted at 2025-10-23 20:17:12 on dist-test-slave-x62p"
server_key: "cfeb7f77208c96e554fd48d036cec2b0"
server_key_iv: "f28a57325f7e81de8e67d4e8f9924eec"
server_key_version: "encryptionkey@0"
I20251023 20:17:12.963359 31170 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:12.974876 31170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:12.980805 31170 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:12.980950 31170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:12.985095 31170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44115
I20251023 20:17:12.985126 31237 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:44115 every 8 connection(s)
I20251023 20:17:12.985479 31170 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:12.986220 31238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:12.989243 31238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5: Bootstrap starting.
I20251023 20:17:12.989894 31238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:12.990020 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31170
I20251023 20:17:12.990154 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:12.990217 31238 log.cc:826] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:12.990357 9687 external_mini_cluster.cc:1468] Setting key e5c1555d0aa6bccf7ed762fa1ce4e89a
I20251023 20:17:12.991206 31238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5: No bootstrap required, opened a new log
I20251023 20:17:12.992944 31238 raft_consensus.cc:359] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417d53cf81ad44d5b59c3ed750c252b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } }
I20251023 20:17:12.993103 31238 raft_consensus.cc:385] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:12.993129 31238 raft_consensus.cc:740] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417d53cf81ad44d5b59c3ed750c252b5, State: Initialized, Role: FOLLOWER
I20251023 20:17:12.993230 31238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417d53cf81ad44d5b59c3ed750c252b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } }
I20251023 20:17:12.993294 31238 raft_consensus.cc:399] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:12.993331 31238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:12.993373 31238 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:12.994051 31238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417d53cf81ad44d5b59c3ed750c252b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } }
I20251023 20:17:12.994190 31238 leader_election.cc:304] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 417d53cf81ad44d5b59c3ed750c252b5; no voters:
I20251023 20:17:12.994360 31238 leader_election.cc:290] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:12.994431 31243 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:12.994606 31238 sys_catalog.cc:565] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:12.995391 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:12.990868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58445 (local address 127.0.0.1:44115)
1023 20:17:12.991103 (+ 235us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:12.991123 (+ 20us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:12.991149 (+ 26us) server_negotiation.cc:408] Connection header received
1023 20:17:12.991630 (+ 481us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:12.991640 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:12.991699 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:12.991868 (+ 169us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:12.992189 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:12.993412 (+ 1223us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:12.994039 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:12.994285 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:12.994767 (+ 482us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:12.994776 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:12.994777 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:12.994970 (+ 193us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:12.994990 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:12.994997 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:12.995069 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:12.995149 (+ 80us) server_negotiation.cc:300] Negotiation successful
1023 20:17:12.995195 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":113,"threads_started":1}
I20251023 20:17:12.996277 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:12.994652 31243 raft_consensus.cc:697] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [term 1 LEADER]: Becoming Leader. State: Replica: 417d53cf81ad44d5b59c3ed750c252b5, State: Running, Role: LEADER
I20251023 20:17:12.996441 31243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417d53cf81ad44d5b59c3ed750c252b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } }
I20251023 20:17:12.996728 31243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 417d53cf81ad44d5b59c3ed750c252b5. Latest consensus state: current_term: 1 leader_uuid: "417d53cf81ad44d5b59c3ed750c252b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417d53cf81ad44d5b59c3ed750c252b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } } }
I20251023 20:17:12.996975 31245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "417d53cf81ad44d5b59c3ed750c252b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417d53cf81ad44d5b59c3ed750c252b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } } }
W20251023 20:17:12.997475 31257 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:12.998003 31257 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:12.997992 31245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:12.997982 31243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:12.998345 31265 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:12.998548 31265 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:13.000582 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:12.997850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54813 (local address 127.0.0.1:44115)
1023 20:17:12.997892 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:12.997894 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:12.998131 (+ 237us) server_negotiation.cc:408] Connection header received
1023 20:17:12.998238 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:12.998240 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:12.998250 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:12.998283 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:12.998663 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:12.999115 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:12.999639 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:12.999841 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:13.000315 (+ 474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:13.000319 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:13.000319 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:13.000324 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:13.000341 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:13.000344 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:13.000392 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:13.000453 (+ 61us) server_negotiation.cc:300] Negotiation successful
1023 20:17:13.000479 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251023 20:17:13.009941 31265 catalog_manager.cc:1357] Generated new cluster ID: 9fdd7d72c3604bf79a7d8ce11343d573
I20251023 20:17:13.010041 31265 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:13.023519 31265 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:13.024351 31265 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:13.034567 31265 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 417d53cf81ad44d5b59c3ed750c252b5: Generated new TSK 0
I20251023 20:17:13.034848 31265 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:13.038371 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31170
2025-10-23T20:17:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:13 dist-test-slave-x62p krb5kdc[31269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:13 dist-test-slave-x62p krb5kdc[31269](info): set up 2 sockets
Oct 23 20:17:13 dist-test-slave-x62p krb5kdc[31269](info): commencing operation
krb5kdc: starting...
W20251023 20:17:15.097373 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.039s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:15 dist-test-slave-x62p krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:15.260605 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:15.347034 31285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:15.347229 31285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:15.347252 31285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:15.348783 31285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:15.348834 31285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:15.348850 31285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:15.348862 31285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:15.348874 31285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:15.350564 31285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:15.350806 31285 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:15.351014 31285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:15.353631 31291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:15.353708 31293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:15.353688 31290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:15.353785 31285 server_base.cc:1047] running on GCE node
I20251023 20:17:15.354058 31285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:15.354276 31285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:15.355427 31285 hybrid_clock.cc:648] HybridClock initialized: now 1761250635355396 us; error 47 us; skew 500 ppm
Oct 23 20:17:15 dist-test-slave-x62p krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:15.358342 31285 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:15.359017 31285 webserver.cc:492] Webserver started at http://127.0.0.1:45223/ using document root <none> and password file <none>
I20251023 20:17:15.359222 31285 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:15.359267 31285 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:15.359385 31285 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:15.360419 31285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "8906a59944b24473a11e11a0222d56cf"
format_stamp: "Formatted at 2025-10-23 20:17:15 on dist-test-slave-x62p"
server_key: "1021512c73fafb119c2608779d4c10ed"
server_key_iv: "f62e966d15a783f06c2a31f54aef2510"
server_key_version: "encryptionkey@0"
I20251023 20:17:15.360790 31285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "8906a59944b24473a11e11a0222d56cf"
format_stamp: "Formatted at 2025-10-23 20:17:15 on dist-test-slave-x62p"
server_key: "1021512c73fafb119c2608779d4c10ed"
server_key_iv: "f62e966d15a783f06c2a31f54aef2510"
server_key_version: "encryptionkey@0"
I20251023 20:17:15.362151 31285 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:15.363193 31300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:15.363505 31285 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:15.363605 31285 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "8906a59944b24473a11e11a0222d56cf"
format_stamp: "Formatted at 2025-10-23 20:17:15 on dist-test-slave-x62p"
server_key: "1021512c73fafb119c2608779d4c10ed"
server_key_iv: "f62e966d15a783f06c2a31f54aef2510"
server_key_version: "encryptionkey@0"
I20251023 20:17:15.363739 31285 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:15.367282 31285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:15.369427 31285 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:15.369553 31285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:15.373600 31285 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40301
I20251023 20:17:15.373668 31352 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:40301 every 8 connection(s)
I20251023 20:17:15.373976 31285 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:15.374622 31353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:15.375231 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31285
I20251023 20:17:15.375362 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:15.375563 9687 external_mini_cluster.cc:1468] Setting key 3a0b7b0659d0d13bb60c225db7663ac7
I20251023 20:17:15.378088 31353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf: Bootstrap starting.
I20251023 20:17:15.378489 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:15.376289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58457 (local address 127.0.0.1:40301)
1023 20:17:15.376638 (+ 349us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:15.376799 (+ 161us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:15.376834 (+ 35us) server_negotiation.cc:408] Connection header received
1023 20:17:15.377172 (+ 338us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:15.377178 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:15.377206 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:15.377364 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:15.378138 (+ 774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:15.378141 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:15.378142 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:15.378148 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:15.378176 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:15.378185 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:15.378273 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:15.378311 (+ 38us) server_negotiation.cc:300] Negotiation successful
1023 20:17:15.378347 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":161,"threads_started":1}
I20251023 20:17:15.378816 31353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:15.379125 31353 log.cc:826] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:15.379606 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:15.379915 31353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf: No bootstrap required, opened a new log
I20251023 20:17:15.381677 31353 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8906a59944b24473a11e11a0222d56cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } }
I20251023 20:17:15.381879 31353 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:15.381915 31353 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8906a59944b24473a11e11a0222d56cf, State: Initialized, Role: FOLLOWER
I20251023 20:17:15.382071 31353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8906a59944b24473a11e11a0222d56cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } }
I20251023 20:17:15.382161 31353 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:15.382225 31353 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:15.382309 31353 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:15.382635 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:15.381204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48155 (local address 127.0.0.1:40301)
1023 20:17:15.381251 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:15.381253 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:15.381330 (+ 77us) server_negotiation.cc:408] Connection header received
1023 20:17:15.381399 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:15.381400 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:15.381406 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:15.381455 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:15.382295 (+ 840us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:15.382297 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:15.382297 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:15.382302 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:15.382320 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:15.382325 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:15.382498 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:15.382535 (+ 37us) server_negotiation.cc:300] Negotiation successful
1023 20:17:15.382553 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:17:15.383381 31353 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8906a59944b24473a11e11a0222d56cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } }
I20251023 20:17:15.383512 31353 leader_election.cc:304] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8906a59944b24473a11e11a0222d56cf; no voters:
I20251023 20:17:15.383690 31353 leader_election.cc:290] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:15.383782 31363 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:15.383919 31353 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:15.383932 31363 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [term 1 LEADER]: Becoming Leader. State: Replica: 8906a59944b24473a11e11a0222d56cf, State: Running, Role: LEADER
I20251023 20:17:15.384054 31363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8906a59944b24473a11e11a0222d56cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } }
I20251023 20:17:15.384393 31364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8906a59944b24473a11e11a0222d56cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8906a59944b24473a11e11a0222d56cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } } }
I20251023 20:17:15.384496 31364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:15.384514 31365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8906a59944b24473a11e11a0222d56cf. Latest consensus state: current_term: 1 leader_uuid: "8906a59944b24473a11e11a0222d56cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8906a59944b24473a11e11a0222d56cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } } }
I20251023 20:17:15.384840 31365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:15.384922 31369 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:15.385593 31369 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:15.387313 31369 catalog_manager.cc:1357] Generated new cluster ID: 55c00254bcea4c09b2603b85728f5ef3
I20251023 20:17:15.387370 31369 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:15.396450 31369 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:15.396672 31369 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:15.400148 31369 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8906a59944b24473a11e11a0222d56cf: Generated new TSK 0
I20251023 20:17:15.400354 31369 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:15.410670 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31285
2025-10-23T20:17:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:15 dist-test-slave-x62p krb5kdc[31384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:15 dist-test-slave-x62p krb5kdc[31384](info): set up 2 sockets
Oct 23 20:17:15 dist-test-slave-x62p krb5kdc[31384](info): commencing operation
krb5kdc: starting...
W20251023 20:17:17.429421 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:17 dist-test-slave-x62p krb5kdc[31384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:17.583411 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:17.669808 31400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:17.670032 31400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:17.670081 31400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:17.671653 31400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:17.671716 31400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:17.671754 31400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:17.671772 31400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:17.671790 31400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:17.673578 31400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:17.673877 31400 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:17.674079 31400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:17.677376 31405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:17.677439 31400 server_base.cc:1047] running on GCE node
W20251023 20:17:17.677433 31408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:17.677379 31406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:17.677867 31400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:17.678158 31400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:17.679332 31400 hybrid_clock.cc:648] HybridClock initialized: now 1761250637679317 us; error 34 us; skew 500 ppm
Oct 23 20:17:17 dist-test-slave-x62p krb5kdc[31384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:17.682241 31400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:17.682884 31400 webserver.cc:492] Webserver started at http://127.0.0.1:46327/ using document root <none> and password file <none>
I20251023 20:17:17.683094 31400 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:17.683136 31400 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:17.683259 31400 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:17.684314 31400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "d97fc366d8c24369a639c3393eefc538"
format_stamp: "Formatted at 2025-10-23 20:17:17 on dist-test-slave-x62p"
server_key: "d1a597700f8c3bb1341999f6e6dfdfba"
server_key_iv: "3b46ebafc8066c1a9c8ef854b8eefb12"
server_key_version: "encryptionkey@0"
I20251023 20:17:17.684649 31400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "d97fc366d8c24369a639c3393eefc538"
format_stamp: "Formatted at 2025-10-23 20:17:17 on dist-test-slave-x62p"
server_key: "d1a597700f8c3bb1341999f6e6dfdfba"
server_key_iv: "3b46ebafc8066c1a9c8ef854b8eefb12"
server_key_version: "encryptionkey@0"
I20251023 20:17:17.686071 31400 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:17.686969 31415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:17.687319 31400 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251023 20:17:17.687419 31400 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "d97fc366d8c24369a639c3393eefc538"
format_stamp: "Formatted at 2025-10-23 20:17:17 on dist-test-slave-x62p"
server_key: "d1a597700f8c3bb1341999f6e6dfdfba"
server_key_iv: "3b46ebafc8066c1a9c8ef854b8eefb12"
server_key_version: "encryptionkey@0"
I20251023 20:17:17.687497 31400 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:17.719705 31400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:17.721850 31400 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:17.722007 31400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:17.726743 31400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33867
I20251023 20:17:17.726838 31467 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:33867 every 8 connection(s)
I20251023 20:17:17.727164 31400 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:17.727870 31468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:17.729074 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31400
I20251023 20:17:17.729205 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:17.729379 9687 external_mini_cluster.cc:1468] Setting key fb8fbd5a25a6119b1e33b3dcccf5f590
I20251023 20:17:17.731498 31468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538: Bootstrap starting.
I20251023 20:17:17.732254 31468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:17.732568 31468 log.cc:826] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:17.733403 31468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538: No bootstrap required, opened a new log
I20251023 20:17:17.734925 31468 raft_consensus.cc:359] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97fc366d8c24369a639c3393eefc538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20251023 20:17:17.735102 31468 raft_consensus.cc:385] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:17.735145 31468 raft_consensus.cc:740] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97fc366d8c24369a639c3393eefc538, State: Initialized, Role: FOLLOWER
I20251023 20:17:17.735263 31468 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97fc366d8c24369a639c3393eefc538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20251023 20:17:17.735332 31468 raft_consensus.cc:399] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:17.735370 31468 raft_consensus.cc:493] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:17.735416 31468 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 0 FOLLOWER]: Advancing to term 1
Oct 23 20:17:17 dist-test-slave-x62p krb5kdc[31384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251023 20:17:17.736317 31468 raft_consensus.cc:515] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97fc366d8c24369a639c3393eefc538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20251023 20:17:17.736452 31468 leader_election.cc:304] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d97fc366d8c24369a639c3393eefc538; no voters:
I20251023 20:17:17.736644 31468 leader_election.cc:290] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:17.736717 31473 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:17.736929 31468 sys_catalog.cc:565] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:17.737439 31473 raft_consensus.cc:697] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [term 1 LEADER]: Becoming Leader. State: Replica: d97fc366d8c24369a639c3393eefc538, State: Running, Role: LEADER
I20251023 20:17:17.737634 31473 consensus_queue.cc:237] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97fc366d8c24369a639c3393eefc538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20251023 20:17:17.739754 31474 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d97fc366d8c24369a639c3393eefc538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97fc366d8c24369a639c3393eefc538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } } }
I20251023 20:17:17.740025 31474 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:17.739751 31475 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d97fc366d8c24369a639c3393eefc538. Latest consensus state: current_term: 1 leader_uuid: "d97fc366d8c24369a639c3393eefc538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97fc366d8c24369a639c3393eefc538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } } }
I20251023 20:17:17.740257 31475 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538 [sys.catalog]: This master's current role is: LEADER
W20251023 20:17:17.740329 31487 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:17.740378 31487 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:17.740460 31489 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:17.740648 31489 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:17.741446 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:17.730025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:52965 (local address 127.0.0.1:33867)
1023 20:17:17.730308 (+ 283us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:17.730320 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:17.730345 (+ 25us) server_negotiation.cc:408] Connection header received
1023 20:17:17.730746 (+ 401us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:17.730755 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:17.730825 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:17.730981 (+ 156us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:17.731789 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:17.732965 (+ 1176us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:17.733646 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:17.733939 (+ 293us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:17.736333 (+ 2394us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:17.736342 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:17.736343 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:17.736363 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:17.739362 (+ 2999us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:17.739875 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:17.739879 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:17.739879 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:17.739943 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:17.740341 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:17.740342 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:17.740343 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:17.740784 (+ 441us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:17.741042 (+ 258us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:17.741146 (+ 104us) server_negotiation.cc:300] Negotiation successful
1023 20:17:17.741259 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":177,"threads_started":1}
I20251023 20:17:17.742277 31489 catalog_manager.cc:1357] Generated new cluster ID: f73beb44210147538ec2a3ebbb3c421c
I20251023 20:17:17.742332 31489 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:17.744818 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:17.750075 31489 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:17.750649 31489 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:17.753834 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:17.746437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:40029 (local address 127.0.0.1:33867)
1023 20:17:17.746488 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:17.746490 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:17.746678 (+ 188us) server_negotiation.cc:408] Connection header received
1023 20:17:17.746753 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:17.746754 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:17.746766 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:17.746828 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:17.747824 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:17.748566 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:17.749372 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:17.749586 (+ 214us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:17.750892 (+ 1306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:17.750897 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:17.750897 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:17.750904 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:17.752603 (+ 1699us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:17.753096 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:17.753098 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:17.753099 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:17.753126 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:17.753323 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:17.753323 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:17.753324 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:17.753431 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:17.753530 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:17.753673 (+ 143us) server_negotiation.cc:300] Negotiation successful
1023 20:17:17.753739 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251023 20:17:17.759163 31489 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d97fc366d8c24369a639c3393eefc538: Generated new TSK 0
I20251023 20:17:17.759434 31489 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:17.766737 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31400
W20251023 20:17:17.771171 31382 connection.cc:537] client connection to 127.0.0.1:33867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33867): Connection reset by peer (error 104)
2025-10-23T20:17:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:17 dist-test-slave-x62p krb5kdc[31499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:17 dist-test-slave-x62p krb5kdc[31499](info): set up 2 sockets
Oct 23 20:17:17 dist-test-slave-x62p krb5kdc[31499](info): commencing operation
krb5kdc: starting...
W20251023 20:17:19.865386 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.080s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:19 dist-test-slave-x62p krb5kdc[31499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:20.022106 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:20.117280 31515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:20.117527 31515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:20.117570 31515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:20.119194 31515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:20.119257 31515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:20.119285 31515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:20.119304 31515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:20.119325 31515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:20.121191 31515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:20.121484 31515 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:20.121685 31515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:20.124476 31521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:20.124516 31520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:20.124485 31523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:20.124871 31515 server_base.cc:1047] running on GCE node
I20251023 20:17:20.125272 31515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:20.125538 31515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:20.126781 31515 hybrid_clock.cc:648] HybridClock initialized: now 1761250640126753 us; error 47 us; skew 500 ppm
Oct 23 20:17:20 dist-test-slave-x62p krb5kdc[31499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:20.129720 31515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:20.130303 31515 webserver.cc:492] Webserver started at http://127.0.0.1:38277/ using document root <none> and password file <none>
I20251023 20:17:20.130486 31515 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:20.130522 31515 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:20.130610 31515 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:20.131579 31515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "eba46240b6994807a437d0da274f6a7a"
format_stamp: "Formatted at 2025-10-23 20:17:20 on dist-test-slave-x62p"
server_key: "b64fc7e341cfe62a644c698b6b8ee932"
server_key_iv: "b7ab1aaefd5d64b4f233685ca4a91125"
server_key_version: "encryptionkey@0"
I20251023 20:17:20.131863 31515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "eba46240b6994807a437d0da274f6a7a"
format_stamp: "Formatted at 2025-10-23 20:17:20 on dist-test-slave-x62p"
server_key: "b64fc7e341cfe62a644c698b6b8ee932"
server_key_iv: "b7ab1aaefd5d64b4f233685ca4a91125"
server_key_version: "encryptionkey@0"
I20251023 20:17:20.133180 31515 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:17:20.133963 31530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:20.134172 31515 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:20.134251 31515 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "eba46240b6994807a437d0da274f6a7a"
format_stamp: "Formatted at 2025-10-23 20:17:20 on dist-test-slave-x62p"
server_key: "b64fc7e341cfe62a644c698b6b8ee932"
server_key_iv: "b7ab1aaefd5d64b4f233685ca4a91125"
server_key_version: "encryptionkey@0"
I20251023 20:17:20.134325 31515 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:20.142603 31515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:20.145391 31515 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:20.145608 31515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:20.150710 31515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40615
I20251023 20:17:20.150792 31582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:40615 every 8 connection(s)
I20251023 20:17:20.151293 31515 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:20.152077 31583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:20.155259 31583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a: Bootstrap starting.
I20251023 20:17:20.156122 31583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:20.156388 31583 log.cc:826] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:20.157207 31583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a: No bootstrap required, opened a new log
I20251023 20:17:20.157873 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31515
I20251023 20:17:20.157987 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:20.158161 9687 external_mini_cluster.cc:1468] Setting key 9c65edc96be5cc004e6643a141a4c318
I20251023 20:17:20.158641 31583 raft_consensus.cc:359] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba46240b6994807a437d0da274f6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } }
I20251023 20:17:20.158871 31583 raft_consensus.cc:385] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:20.158926 31583 raft_consensus.cc:740] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba46240b6994807a437d0da274f6a7a, State: Initialized, Role: FOLLOWER
I20251023 20:17:20.159039 31583 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba46240b6994807a437d0da274f6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } }
I20251023 20:17:20.159114 31583 raft_consensus.cc:399] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:20.159159 31583 raft_consensus.cc:493] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:20.159215 31583 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:20.159958 31583 raft_consensus.cc:515] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba46240b6994807a437d0da274f6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } }
I20251023 20:17:20.160070 31583 leader_election.cc:304] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba46240b6994807a437d0da274f6a7a; no voters:
I20251023 20:17:20.160230 31583 leader_election.cc:290] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:20.160290 31588 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:20.160410 31588 raft_consensus.cc:697] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [term 1 LEADER]: Becoming Leader. State: Replica: eba46240b6994807a437d0da274f6a7a, State: Running, Role: LEADER
I20251023 20:17:20.160497 31583 sys_catalog.cc:565] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:20.160581 31588 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba46240b6994807a437d0da274f6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } }
I20251023 20:17:20.161007 31588 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba46240b6994807a437d0da274f6a7a. Latest consensus state: current_term: 1 leader_uuid: "eba46240b6994807a437d0da274f6a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba46240b6994807a437d0da274f6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } } }
I20251023 20:17:20.161023 31590 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba46240b6994807a437d0da274f6a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba46240b6994807a437d0da274f6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } } }
I20251023 20:17:20.161183 31590 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:20.161396 31588 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:20.161962 31596 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:20.162537 31596 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:20.164042 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:20.158707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:40803 (local address 127.0.0.1:40615)
1023 20:17:20.158888 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:20.158901 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:20.158928 (+ 27us) server_negotiation.cc:408] Connection header received
1023 20:17:20.159049 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:20.159053 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:20.159096 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:20.159228 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:20.159682 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:20.160677 (+ 995us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:20.162050 (+ 1373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:20.162572 (+ 522us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:20.163135 (+ 563us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:20.163142 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:20.163142 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:20.163359 (+ 217us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:20.163384 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:20.163393 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:20.163503 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:20.163842 (+ 339us) server_negotiation.cc:300] Negotiation successful
1023 20:17:20.163871 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20251023 20:17:20.164414 31596 catalog_manager.cc:1357] Generated new cluster ID: 05819b6049184b48a53ca3f71eced82b
I20251023 20:17:20.164472 31596 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:20.165127 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:20.170238 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:20.166863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:35427 (local address 127.0.0.1:40615)
1023 20:17:20.166926 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:20.166929 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:20.166976 (+ 47us) server_negotiation.cc:408] Connection header received
1023 20:17:20.167059 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:20.167060 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:20.167071 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:20.167138 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:20.167491 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:20.168206 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:20.169023 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:20.169196 (+ 173us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:20.169981 (+ 785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:20.169984 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:20.169985 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:20.169990 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:20.170003 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:20.170006 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:20.170051 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:20.170136 (+ 85us) server_negotiation.cc:300] Negotiation successful
1023 20:17:20.170147 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251023 20:17:20.177001 31596 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:20.177601 31596 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:20.187407 31596 catalog_manager.cc:6022] T 00000000000000000000000000000000 P eba46240b6994807a437d0da274f6a7a: Generated new TSK 0
I20251023 20:17:20.187631 31596 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:20.190438 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31515
W20251023 20:17:20.194707 31497 connection.cc:537] client connection to 127.0.0.1:40615 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40615): Connection reset by peer (error 104)
2025-10-23T20:17:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:20 dist-test-slave-x62p krb5kdc[31614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:20 dist-test-slave-x62p krb5kdc[31614](info): set up 2 sockets
Oct 23 20:17:20 dist-test-slave-x62p krb5kdc[31614](info): commencing operation
krb5kdc: starting...
W20251023 20:17:22.297577 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:22 dist-test-slave-x62p krb5kdc[31614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:22.479656 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:22.567850 31630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:22.568096 31630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:22.568126 31630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:22.569806 31630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:22.569883 31630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:22.569900 31630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:22.569912 31630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:22.569924 31630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:22.571779 31630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:22.572036 31630 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:22.572209 31630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:22.575152 31636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:22.575310 31635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:22.575403 31638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:22.575464 31630 server_base.cc:1047] running on GCE node
I20251023 20:17:22.575701 31630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:22.575940 31630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:22.577091 31630 hybrid_clock.cc:648] HybridClock initialized: now 1761250642577067 us; error 41 us; skew 500 ppm
Oct 23 20:17:22 dist-test-slave-x62p krb5kdc[31614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:22.579947 31630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:22.580605 31630 webserver.cc:492] Webserver started at http://127.0.0.1:36777/ using document root <none> and password file <none>
I20251023 20:17:22.580852 31630 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:22.580894 31630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:22.581017 31630 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:22.582078 31630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "9860e70d290e4d9595cb7f21ca57b7bb"
format_stamp: "Formatted at 2025-10-23 20:17:22 on dist-test-slave-x62p"
server_key: "2df3540514ebf3033f0fb56c6a7647bc"
server_key_iv: "75dff183c4940a60d3813aaaa14311cc"
server_key_version: "encryptionkey@0"
I20251023 20:17:22.582414 31630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "9860e70d290e4d9595cb7f21ca57b7bb"
format_stamp: "Formatted at 2025-10-23 20:17:22 on dist-test-slave-x62p"
server_key: "2df3540514ebf3033f0fb56c6a7647bc"
server_key_iv: "75dff183c4940a60d3813aaaa14311cc"
server_key_version: "encryptionkey@0"
I20251023 20:17:22.583860 31630 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:17:22.584702 31645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:22.585000 31630 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:22.585086 31630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "9860e70d290e4d9595cb7f21ca57b7bb"
format_stamp: "Formatted at 2025-10-23 20:17:22 on dist-test-slave-x62p"
server_key: "2df3540514ebf3033f0fb56c6a7647bc"
server_key_iv: "75dff183c4940a60d3813aaaa14311cc"
server_key_version: "encryptionkey@0"
I20251023 20:17:22.585160 31630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:22.589447 31630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:22.592140 31630 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:22.592304 31630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:22.599022 31630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42133
I20251023 20:17:22.599215 31697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42133 every 8 connection(s)
I20251023 20:17:22.599460 31630 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:22.600260 31698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:22.603917 31698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb: Bootstrap starting.
I20251023 20:17:22.604789 31698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:22.604893 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31630
I20251023 20:17:22.605037 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:22.605160 31698 log.cc:826] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:22.605242 9687 external_mini_cluster.cc:1468] Setting key 07d97e2f3ec1d92915259f46405c6d96
I20251023 20:17:22.606544 31698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb: No bootstrap required, opened a new log
I20251023 20:17:22.609117 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:22.606401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:50215 (local address 127.0.0.1:42133)
1023 20:17:22.606597 (+ 196us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:22.606611 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:22.606636 (+ 25us) server_negotiation.cc:408] Connection header received
1023 20:17:22.607087 (+ 451us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:22.607097 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:22.607134 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:22.607817 (+ 683us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:22.608402 (+ 585us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:22.608406 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:22.608406 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:22.608411 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:22.608433 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:22.608441 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:22.608838 (+ 397us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:22.608887 (+ 49us) server_negotiation.cc:300] Negotiation successful
1023 20:17:22.608930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20251023 20:17:22.609515 31698 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9860e70d290e4d9595cb7f21ca57b7bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42133 } }
I20251023 20:17:22.609671 31698 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:22.609694 31698 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9860e70d290e4d9595cb7f21ca57b7bb, State: Initialized, Role: FOLLOWER
I20251023 20:17:22.609812 31698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9860e70d290e4d9595cb7f21ca57b7bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42133 } }
I20251023 20:17:22.609892 31698 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:22.609921 31698 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:22.609982 31698 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:22.610188 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:22.610872 31698 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9860e70d290e4d9595cb7f21ca57b7bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42133 } }
I20251023 20:17:22.611027 31698 leader_election.cc:304] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9860e70d290e4d9595cb7f21ca57b7bb; no voters:
I20251023 20:17:22.611207 31698 leader_election.cc:290] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:22.611289 31703 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:22.611474 31703 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [term 1 LEADER]: Becoming Leader. State: Replica: 9860e70d290e4d9595cb7f21ca57b7bb, State: Running, Role: LEADER
I20251023 20:17:22.611503 31698 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:22.611618 31703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9860e70d290e4d9595cb7f21ca57b7bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42133 } }
I20251023 20:17:22.612152 31708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9860e70d290e4d9595cb7f21ca57b7bb. Latest consensus state: current_term: 1 leader_uuid: "9860e70d290e4d9595cb7f21ca57b7bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9860e70d290e4d9595cb7f21ca57b7bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42133 } } }
I20251023 20:17:22.612279 31708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:22.612229 31704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9860e70d290e4d9595cb7f21ca57b7bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9860e70d290e4d9595cb7f21ca57b7bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42133 } } }
I20251023 20:17:22.612427 31704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:22.612958 31715 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:22.613580 31715 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:22.615021 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:22.612016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:60179 (local address 127.0.0.1:42133)
1023 20:17:22.612067 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:22.612178 (+ 111us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:22.613413 (+ 1235us) server_negotiation.cc:408] Connection header received
1023 20:17:22.613568 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:22.613569 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:22.613575 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:22.613676 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:22.614642 (+ 966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:22.614643 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:22.614644 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:22.614648 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:22.614665 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:22.614669 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:22.614729 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:22.614940 (+ 211us) server_negotiation.cc:300] Negotiation successful
1023 20:17:22.614958 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:17:22.615762 31715 catalog_manager.cc:1357] Generated new cluster ID: f3007631fd5143b6a16f94ec8b72a1a4
I20251023 20:17:22.615988 31715 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:22.647060 31715 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:22.647282 31715 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:22.654812 31715 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9860e70d290e4d9595cb7f21ca57b7bb: Generated new TSK 0
I20251023 20:17:22.655165 31715 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:22.659822 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31630
2025-10-23T20:17:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:22 dist-test-slave-x62p krb5kdc[31729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:22 dist-test-slave-x62p krb5kdc[31729](info): set up 2 sockets
Oct 23 20:17:22 dist-test-slave-x62p krb5kdc[31729](info): commencing operation
krb5kdc: starting...
W20251023 20:17:24.693377 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:24 dist-test-slave-x62p krb5kdc[31729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:24.849123 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:24.943907 31745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:24.944145 31745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:24.944193 31745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:24.945961 31745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:24.946035 31745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:24.946059 31745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:24.946084 31745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:24.946103 31745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:24.948091 31745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:24.948406 31745 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:24.948613 31745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:24.952586 31750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:24.952574 31753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:24.952574 31751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:24.953270 31745 server_base.cc:1047] running on GCE node
I20251023 20:17:24.953557 31745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:24.953994 31745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:24.955174 31745 hybrid_clock.cc:648] HybridClock initialized: now 1761250644955156 us; error 32 us; skew 500 ppm
Oct 23 20:17:24 dist-test-slave-x62p krb5kdc[31729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:24.958876 31745 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:24.959643 31745 webserver.cc:492] Webserver started at http://127.0.0.1:45679/ using document root <none> and password file <none>
I20251023 20:17:24.959878 31745 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:24.959926 31745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:24.960057 31745 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:24.961577 31745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "78f816cdb7a149b58e5db27b3a64a793"
format_stamp: "Formatted at 2025-10-23 20:17:24 on dist-test-slave-x62p"
server_key: "36f3aa983da25a239eddc22712f3279c"
server_key_iv: "70c4617f46989318ab79b25d65ea213f"
server_key_version: "encryptionkey@0"
I20251023 20:17:24.962067 31745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "78f816cdb7a149b58e5db27b3a64a793"
format_stamp: "Formatted at 2025-10-23 20:17:24 on dist-test-slave-x62p"
server_key: "36f3aa983da25a239eddc22712f3279c"
server_key_iv: "70c4617f46989318ab79b25d65ea213f"
server_key_version: "encryptionkey@0"
I20251023 20:17:24.963513 31745 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:17:24.964437 31760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:24.964690 31745 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:24.964829 31745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "78f816cdb7a149b58e5db27b3a64a793"
format_stamp: "Formatted at 2025-10-23 20:17:24 on dist-test-slave-x62p"
server_key: "36f3aa983da25a239eddc22712f3279c"
server_key_iv: "70c4617f46989318ab79b25d65ea213f"
server_key_version: "encryptionkey@0"
I20251023 20:17:24.964955 31745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:24.999936 31745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:25.002544 31745 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:25.002830 31745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:25.007705 31745 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40663
I20251023 20:17:25.007795 31812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:40663 every 8 connection(s)
I20251023 20:17:25.008090 31745 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:25.008769 31813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:25.011560 31813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793: Bootstrap starting.
I20251023 20:17:25.012195 31813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:25.012420 31813 log.cc:826] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:25.013406 31813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793: No bootstrap required, opened a new log
I20251023 20:17:25.014708 31813 raft_consensus.cc:359] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f816cdb7a149b58e5db27b3a64a793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20251023 20:17:25.014869 31813 raft_consensus.cc:385] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:25.014892 31813 raft_consensus.cc:740] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f816cdb7a149b58e5db27b3a64a793, State: Initialized, Role: FOLLOWER
I20251023 20:17:25.014999 31813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f816cdb7a149b58e5db27b3a64a793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20251023 20:17:25.015054 31813 raft_consensus.cc:399] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:25.015089 31813 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:25.015137 31813 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:25.015344 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31745
I20251023 20:17:25.015460 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:25.015632 9687 external_mini_cluster.cc:1468] Setting key 1cd980b217887009b4f7e80d38d90db6
I20251023 20:17:25.016090 31813 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f816cdb7a149b58e5db27b3a64a793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20251023 20:17:25.016234 31813 leader_election.cc:304] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78f816cdb7a149b58e5db27b3a64a793; no voters:
I20251023 20:17:25.016398 31813 leader_election.cc:290] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:25.016448 31818 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:25.016582 31818 raft_consensus.cc:697] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [term 1 LEADER]: Becoming Leader. State: Replica: 78f816cdb7a149b58e5db27b3a64a793, State: Running, Role: LEADER
I20251023 20:17:25.016661 31813 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:25.016682 31818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f816cdb7a149b58e5db27b3a64a793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20251023 20:17:25.017176 31819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78f816cdb7a149b58e5db27b3a64a793. Latest consensus state: current_term: 1 leader_uuid: "78f816cdb7a149b58e5db27b3a64a793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f816cdb7a149b58e5db27b3a64a793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } } }
I20251023 20:17:25.017452 31819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:25.017846 31820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78f816cdb7a149b58e5db27b3a64a793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f816cdb7a149b58e5db27b3a64a793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } } }
I20251023 20:17:25.017930 31820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:25.018026 31823 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:25.019057 31823 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 23 20:17:25 dist-test-slave-x62p krb5kdc[31729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251023 20:17:25.022046 31823 catalog_manager.cc:1357] Generated new cluster ID: 3ee4163a9a8846489fb01eb81b1077dd
I20251023 20:17:25.022100 31823 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:25.025832 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:25.016113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:47051 (local address 127.0.0.1:40663)
1023 20:17:25.016337 (+ 224us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:25.016353 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:25.016387 (+ 34us) server_negotiation.cc:408] Connection header received
1023 20:17:25.016525 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:25.016542 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:25.016599 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:25.016776 (+ 177us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:25.017698 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:25.018973 (+ 1275us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:25.019655 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:25.019950 (+ 295us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:25.022429 (+ 2479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:25.022456 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:25.022456 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:25.022481 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:25.024299 (+ 1818us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:25.024712 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:25.024716 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:25.024716 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:25.024805 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:25.025005 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:25.025006 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:25.025007 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:25.025212 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:25.025355 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:25.025545 (+ 190us) server_negotiation.cc:300] Negotiation successful
1023 20:17:25.025666 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20251023 20:17:25.026846 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:25.034651 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:25.028764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:46309 (local address 127.0.0.1:40663)
1023 20:17:25.028805 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:25.028807 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:25.028927 (+ 120us) server_negotiation.cc:408] Connection header received
1023 20:17:25.028992 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:25.028993 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:25.029005 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:25.029057 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:25.029700 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:25.030179 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:25.030694 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:25.030894 (+ 200us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:25.032011 (+ 1117us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:25.032014 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:25.032015 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:25.032023 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:25.033442 (+ 1419us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:25.033935 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:25.033937 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:25.033937 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:25.033960 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:25.034135 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:25.034136 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:25.034136 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:25.034225 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:25.034325 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:25.034458 (+ 133us) server_negotiation.cc:300] Negotiation successful
1023 20:17:25.034535 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251023 20:17:25.037770 31823 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:25.038383 31823 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:25.043342 31823 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 78f816cdb7a149b58e5db27b3a64a793: Generated new TSK 0
I20251023 20:17:25.043537 31823 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:25.046620 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31745
W20251023 20:17:25.051640 31727 connection.cc:537] client connection to 127.0.0.1:40663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40663): Connection reset by peer (error 104)
2025-10-23T20:17:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:25 dist-test-slave-x62p krb5kdc[31844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:25 dist-test-slave-x62p krb5kdc[31844](info): set up 2 sockets
Oct 23 20:17:25 dist-test-slave-x62p krb5kdc[31844](info): commencing operation
krb5kdc: starting...
W20251023 20:17:27.049197 9687 mini_kdc.cc:121] Time spent starting KDC: real 1.983s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:27 dist-test-slave-x62p krb5kdc[31844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:27.193219 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:27.283743 31860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:27.283985 31860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:27.284039 31860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:27.285689 31860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:27.285756 31860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:27.285786 31860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:27.285811 31860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:27.285831 31860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:27.287869 31860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:27.288182 31860 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:27.288385 31860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:17:27.291750 31860 server_base.cc:1047] running on GCE node
W20251023 20:17:27.291676 31866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:27.291810 31868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:27.291828 31865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:27.292136 31860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:27.292385 31860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:27.293552 31860 hybrid_clock.cc:648] HybridClock initialized: now 1761250647293536 us; error 36 us; skew 500 ppm
Oct 23 20:17:27 dist-test-slave-x62p krb5kdc[31844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:27.296236 31860 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:27.296901 31860 webserver.cc:492] Webserver started at http://127.0.0.1:46507/ using document root <none> and password file <none>
I20251023 20:17:27.297112 31860 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:27.297152 31860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:27.297268 31860 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:27.298280 31860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "097264747e634e319a2418d20a12d603"
format_stamp: "Formatted at 2025-10-23 20:17:27 on dist-test-slave-x62p"
server_key: "7715576b1a90fcf56ab8ace163471ecb"
server_key_iv: "727c15de96d47c4f7b8d85dab1c0e59f"
server_key_version: "encryptionkey@0"
I20251023 20:17:27.298596 31860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "097264747e634e319a2418d20a12d603"
format_stamp: "Formatted at 2025-10-23 20:17:27 on dist-test-slave-x62p"
server_key: "7715576b1a90fcf56ab8ace163471ecb"
server_key_iv: "727c15de96d47c4f7b8d85dab1c0e59f"
server_key_version: "encryptionkey@0"
I20251023 20:17:27.299934 31860 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:17:27.300709 31875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:27.300944 31860 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:27.301028 31860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "097264747e634e319a2418d20a12d603"
format_stamp: "Formatted at 2025-10-23 20:17:27 on dist-test-slave-x62p"
server_key: "7715576b1a90fcf56ab8ace163471ecb"
server_key_iv: "727c15de96d47c4f7b8d85dab1c0e59f"
server_key_version: "encryptionkey@0"
I20251023 20:17:27.301096 31860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:27.318897 31860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:27.321054 31860 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:27.321210 31860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:27.325894 31860 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35771
I20251023 20:17:27.325935 31927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:35771 every 8 connection(s)
I20251023 20:17:27.326278 31860 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:27.327052 31928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:27.328559 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31860
I20251023 20:17:27.328709 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:27.328936 9687 external_mini_cluster.cc:1468] Setting key 5d3f7d4130bad6df409286cb496d34e1
I20251023 20:17:27.330188 31928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603: Bootstrap starting.
I20251023 20:17:27.330901 31928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:27.331146 31928 log.cc:826] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:27.331893 31928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603: No bootstrap required, opened a new log
I20251023 20:17:27.333375 31928 raft_consensus.cc:359] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097264747e634e319a2418d20a12d603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35771 } }
I20251023 20:17:27.333601 31928 raft_consensus.cc:385] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:27.333671 31928 raft_consensus.cc:740] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097264747e634e319a2418d20a12d603, State: Initialized, Role: FOLLOWER
I20251023 20:17:27.333796 31928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097264747e634e319a2418d20a12d603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35771 } }
I20251023 20:17:27.333832 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:27.329498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:48329 (local address 127.0.0.1:35771)
1023 20:17:27.329665 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:27.329675 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:27.329695 (+ 20us) server_negotiation.cc:408] Connection header received
1023 20:17:27.330033 (+ 338us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:27.330039 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:27.330082 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:27.330196 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:27.330652 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:27.331492 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:27.332318 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:27.332515 (+ 197us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:27.333092 (+ 577us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:27.333099 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:27.333099 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:27.333292 (+ 193us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:27.333312 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:27.333319 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:27.333394 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:27.333504 (+ 110us) server_negotiation.cc:300] Negotiation successful
1023 20:17:27.333532 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20251023 20:17:27.333884 31928 raft_consensus.cc:399] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:27.333925 31928 raft_consensus.cc:493] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:27.333969 31928 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:27.334661 31928 raft_consensus.cc:515] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097264747e634e319a2418d20a12d603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35771 } }
I20251023 20:17:27.334774 31928 leader_election.cc:304] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 097264747e634e319a2418d20a12d603; no voters:
I20251023 20:17:27.334864 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:27.334928 31928 leader_election.cc:290] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:27.334978 31933 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:27.335117 31933 raft_consensus.cc:697] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [term 1 LEADER]: Becoming Leader. State: Replica: 097264747e634e319a2418d20a12d603, State: Running, Role: LEADER
I20251023 20:17:27.335194 31928 sys_catalog.cc:565] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:27.335261 31933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097264747e634e319a2418d20a12d603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35771 } }
I20251023 20:17:27.335760 31934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "097264747e634e319a2418d20a12d603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097264747e634e319a2418d20a12d603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35771 } } }
I20251023 20:17:27.335856 31934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:27.335985 31934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 097264747e634e319a2418d20a12d603. Latest consensus state: current_term: 1 leader_uuid: "097264747e634e319a2418d20a12d603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097264747e634e319a2418d20a12d603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35771 } } }
I20251023 20:17:27.336030 31934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:27.336505 31937 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:27.337103 31937 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:27.338744 31937 catalog_manager.cc:1357] Generated new cluster ID: dd6fb5d8d2484acd9fdd0f75dfb7364c
I20251023 20:17:27.338796 31937 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:27.339358 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:27.336764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:44129 (local address 127.0.0.1:35771)
1023 20:17:27.336810 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:27.336813 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:27.336825 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:17:27.336851 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:27.336852 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:27.336862 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:27.336927 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:27.337382 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:27.337867 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:27.338377 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:27.338573 (+ 196us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:27.339073 (+ 500us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:27.339076 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:27.339076 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:27.339081 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:27.339097 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:27.339102 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:27.339169 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:27.339274 (+ 105us) server_negotiation.cc:300] Negotiation successful
1023 20:17:27.339288 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:17:27.398504 31937 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:27.399370 31937 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:27.408641 31937 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 097264747e634e319a2418d20a12d603: Generated new TSK 0
I20251023 20:17:27.409018 31937 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:27.422351 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31860
W20251023 20:17:27.427762 31842 connection.cc:537] client connection to 127.0.0.1:35771 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35771): Connection reset by peer (error 104)
2025-10-23T20:17:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:27 dist-test-slave-x62p krb5kdc[31959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:27 dist-test-slave-x62p krb5kdc[31959](info): set up 2 sockets
Oct 23 20:17:27 dist-test-slave-x62p krb5kdc[31959](info): commencing operation
krb5kdc: starting...
W20251023 20:17:29.474303 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:29 dist-test-slave-x62p krb5kdc[31959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:29.620954 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:29.705739 31975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:29.705943 31975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:29.705971 31975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:29.707489 31975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:29.707547 31975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:29.707563 31975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:29.707576 31975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:29.707588 31975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:29.709261 31975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:29.709498 31975 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:29.709693 31975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:29.712383 31983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:29.712373 31980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:29.712422 31975 server_base.cc:1047] running on GCE node
W20251023 20:17:29.712371 31981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:29.712774 31975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:29.713035 31975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:29.714162 31975 hybrid_clock.cc:648] HybridClock initialized: now 1761250649714146 us; error 37 us; skew 500 ppm
Oct 23 20:17:29 dist-test-slave-x62p krb5kdc[31959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:29.716662 31975 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251023 20:17:29.717259 31975 webserver.cc:492] Webserver started at http://127.0.0.1:33985/ using document root <none> and password file <none>
I20251023 20:17:29.717442 31975 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:29.717474 31975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:29.717571 31975 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:29.718545 31975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "ecaeb3f5012b4c45b6c27f080c04fec1"
format_stamp: "Formatted at 2025-10-23 20:17:29 on dist-test-slave-x62p"
server_key: "26c3cd07674ff47436d797a535e51494"
server_key_iv: "4a21555524f09dfb75ac083812c21ba3"
server_key_version: "encryptionkey@0"
I20251023 20:17:29.718822 31975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "ecaeb3f5012b4c45b6c27f080c04fec1"
format_stamp: "Formatted at 2025-10-23 20:17:29 on dist-test-slave-x62p"
server_key: "26c3cd07674ff47436d797a535e51494"
server_key_iv: "4a21555524f09dfb75ac083812c21ba3"
server_key_version: "encryptionkey@0"
I20251023 20:17:29.720073 31975 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251023 20:17:29.720911 31990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:29.721096 31975 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:29.721170 31975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "ecaeb3f5012b4c45b6c27f080c04fec1"
format_stamp: "Formatted at 2025-10-23 20:17:29 on dist-test-slave-x62p"
server_key: "26c3cd07674ff47436d797a535e51494"
server_key_iv: "4a21555524f09dfb75ac083812c21ba3"
server_key_version: "encryptionkey@0"
I20251023 20:17:29.721230 31975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:29.724807 31975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:29.731158 31975 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:29.731287 31975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:29.735728 31975 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45519
I20251023 20:17:29.735885 32042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:45519 every 8 connection(s)
I20251023 20:17:29.736127 31975 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:29.736867 32043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:29.739921 32043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1: Bootstrap starting.
I20251023 20:17:29.741005 32043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:29.741276 32043 log.cc:826] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:29.742702 32043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1: No bootstrap required, opened a new log
I20251023 20:17:29.744473 32043 raft_consensus.cc:359] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaeb3f5012b4c45b6c27f080c04fec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } }
I20251023 20:17:29.744632 32043 raft_consensus.cc:385] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:29.744655 32043 raft_consensus.cc:740] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaeb3f5012b4c45b6c27f080c04fec1, State: Initialized, Role: FOLLOWER
I20251023 20:17:29.744829 32043 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaeb3f5012b4c45b6c27f080c04fec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } }
I20251023 20:17:29.744913 32043 raft_consensus.cc:399] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:29.744938 32043 raft_consensus.cc:493] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:29.744969 32043 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:29.745596 32043 raft_consensus.cc:515] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaeb3f5012b4c45b6c27f080c04fec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } }
I20251023 20:17:29.745713 32043 leader_election.cc:304] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecaeb3f5012b4c45b6c27f080c04fec1; no voters:
I20251023 20:17:29.745872 32043 leader_election.cc:290] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:29.745925 32046 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:29.746086 32046 raft_consensus.cc:697] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [term 1 LEADER]: Becoming Leader. State: Replica: ecaeb3f5012b4c45b6c27f080c04fec1, State: Running, Role: LEADER
I20251023 20:17:29.746094 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 31975
I20251023 20:17:29.746161 32043 sys_catalog.cc:565] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:29.746232 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:29.746225 32046 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaeb3f5012b4c45b6c27f080c04fec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } }
I20251023 20:17:29.746435 9687 external_mini_cluster.cc:1468] Setting key 0ce9e72d4d65de5e1cfdbd8f1fcf3ebe
I20251023 20:17:29.746580 32047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecaeb3f5012b4c45b6c27f080c04fec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaeb3f5012b4c45b6c27f080c04fec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } } }
I20251023 20:17:29.746645 32047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:29.746768 32047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecaeb3f5012b4c45b6c27f080c04fec1. Latest consensus state: current_term: 1 leader_uuid: "ecaeb3f5012b4c45b6c27f080c04fec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaeb3f5012b4c45b6c27f080c04fec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } } }
I20251023 20:17:29.746819 32047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:29.747524 32051 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:29.748098 32051 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:29.749807 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:29.747085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:44503 (local address 127.0.0.1:45519)
1023 20:17:29.748248 (+ 1163us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:29.748262 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:29.748288 (+ 26us) server_negotiation.cc:408] Connection header received
1023 20:17:29.748413 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:29.748420 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:29.748456 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:29.748605 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:29.749421 (+ 816us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:29.749425 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:29.749425 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:29.749431 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:29.749458 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:29.749466 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:29.749564 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:29.749603 (+ 39us) server_negotiation.cc:300] Negotiation successful
1023 20:17:29.749641 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":84,"threads_started":1}
I20251023 20:17:29.750859 32051 catalog_manager.cc:1357] Generated new cluster ID: a7e199d74a8b4df9b4e7de8d723d97e3
I20251023 20:17:29.750873 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:29.750924 32051 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:29.754132 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:29.752761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:40957 (local address 127.0.0.1:45519)
1023 20:17:29.752830 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:29.752833 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:29.752967 (+ 134us) server_negotiation.cc:408] Connection header received
1023 20:17:29.753024 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:29.753025 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:29.753031 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:29.753101 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:29.753886 (+ 785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:29.753888 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:29.753888 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:29.753894 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:29.753915 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:29.753919 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:29.753995 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:29.754039 (+ 44us) server_negotiation.cc:300] Negotiation successful
1023 20:17:29.754054 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251023 20:17:29.760305 32051 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:29.760490 32051 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:29.783772 32051 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ecaeb3f5012b4c45b6c27f080c04fec1: Generated new TSK 0
I20251023 20:17:29.784011 32051 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:29.795977 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 31975
2025-10-23T20:17:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:29 dist-test-slave-x62p krb5kdc[32074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:29 dist-test-slave-x62p krb5kdc[32074](info): set up 2 sockets
Oct 23 20:17:29 dist-test-slave-x62p krb5kdc[32074](info): commencing operation
krb5kdc: starting...
W20251023 20:17:31.825235 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:31 dist-test-slave-x62p krb5kdc[32074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:31.972184 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37657
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:32.055464 32090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:32.055703 32090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:32.055760 32090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:32.057511 32090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:32.057585 32090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:32.057617 32090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:32.057637 32090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:32.057657 32090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:32.059532 32090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:32.059836 32090 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:32.060040 32090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:32.062932 32098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:32.062942 32096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:32.062943 32095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:32.063001 32090 server_base.cc:1047] running on GCE node
I20251023 20:17:32.063402 32090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:32.063632 32090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:32.064778 32090 hybrid_clock.cc:648] HybridClock initialized: now 1761250652064761 us; error 35 us; skew 500 ppm
Oct 23 20:17:32 dist-test-slave-x62p krb5kdc[32074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250652, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:32.067329 32090 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:17:32.067873 32090 webserver.cc:492] Webserver started at http://127.9.117.254:43013/ using document root <none> and password file <none>
I20251023 20:17:32.068063 32090 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:32.068104 32090 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:32.068215 32090 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:32.069259 32090 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "4c59668051c14949a0b8b90b3cd6b122"
format_stamp: "Formatted at 2025-10-23 20:17:32 on dist-test-slave-x62p"
server_key: "2bc8d2d91a68084fe17c6ba5f9a84c34"
server_key_iv: "2322bcf501c8a453281fecbf107e7e15"
server_key_version: "encryptionkey@0"
I20251023 20:17:32.069594 32090 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "4c59668051c14949a0b8b90b3cd6b122"
format_stamp: "Formatted at 2025-10-23 20:17:32 on dist-test-slave-x62p"
server_key: "2bc8d2d91a68084fe17c6ba5f9a84c34"
server_key_iv: "2322bcf501c8a453281fecbf107e7e15"
server_key_version: "encryptionkey@0"
I20251023 20:17:32.070910 32090 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:17:32.071686 32105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:32.071902 32090 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:32.071981 32090 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "4c59668051c14949a0b8b90b3cd6b122"
format_stamp: "Formatted at 2025-10-23 20:17:32 on dist-test-slave-x62p"
server_key: "2bc8d2d91a68084fe17c6ba5f9a84c34"
server_key_iv: "2322bcf501c8a453281fecbf107e7e15"
server_key_version: "encryptionkey@0"
I20251023 20:17:32.072045 32090 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:32.079696 32090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:32.081683 32090 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:32.081811 32090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:32.086279 32090 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37657
I20251023 20:17:32.086324 32157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37657 every 8 connection(s)
I20251023 20:17:32.086647 32090 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:32.087129 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 32090
I20251023 20:17:32.087272 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:32.087442 32158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:32.087493 9687 external_mini_cluster.cc:1468] Setting key 01e2f8f330422265cb56418fd382661e
I20251023 20:17:32.091403 32158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122: Bootstrap starting.
I20251023 20:17:32.092303 32158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:32.092650 32158 log.cc:826] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:32.093580 32158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122: No bootstrap required, opened a new log
Oct 23 20:17:32 dist-test-slave-x62p krb5kdc[32074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:32.094741 32158 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c59668051c14949a0b8b90b3cd6b122" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37657 } }
I20251023 20:17:32.094875 32158 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:32.094907 32158 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c59668051c14949a0b8b90b3cd6b122, State: Initialized, Role: FOLLOWER
I20251023 20:17:32.095029 32158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c59668051c14949a0b8b90b3cd6b122" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37657 } }
I20251023 20:17:32.095089 32158 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:32.095135 32158 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:32.095191 32158 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:32.095947 32158 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c59668051c14949a0b8b90b3cd6b122" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37657 } }
I20251023 20:17:32.096097 32158 leader_election.cc:304] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c59668051c14949a0b8b90b3cd6b122; no voters:
I20251023 20:17:32.096274 32158 leader_election.cc:290] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:32.096351 32163 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:32.096552 32158 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:32.096575 32163 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [term 1 LEADER]: Becoming Leader. State: Replica: 4c59668051c14949a0b8b90b3cd6b122, State: Running, Role: LEADER
I20251023 20:17:32.096822 32163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c59668051c14949a0b8b90b3cd6b122" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37657 } }
I20251023 20:17:32.097365 32165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c59668051c14949a0b8b90b3cd6b122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c59668051c14949a0b8b90b3cd6b122" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37657 } } }
I20251023 20:17:32.097473 32165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:32.097622 32165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c59668051c14949a0b8b90b3cd6b122. Latest consensus state: current_term: 1 leader_uuid: "4c59668051c14949a0b8b90b3cd6b122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c59668051c14949a0b8b90b3cd6b122" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37657 } } }
I20251023 20:17:32.097667 32165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122 [sys.catalog]: This master's current role is: LEADER
W20251023 20:17:32.098732 32179 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:32.098799 32179 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:32.098846 32172 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:32.098996 32172 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:32.099301 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:32.088042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48461 (local address 127.9.117.254:37657)
1023 20:17:32.088192 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:32.088203 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:32.088223 (+ 20us) server_negotiation.cc:408] Connection header received
1023 20:17:32.088609 (+ 386us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:32.088615 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:32.088658 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:32.088874 (+ 216us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:32.089979 (+ 1105us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:32.090783 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:32.091721 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:32.091925 (+ 204us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:32.094613 (+ 2688us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:32.094624 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:32.094624 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:32.094692 (+ 68us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:32.097104 (+ 2412us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:32.097717 (+ 613us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:32.097720 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:32.097721 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:32.097800 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:32.098082 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:32.098083 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:32.098083 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:32.098403 (+ 320us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:32.098656 (+ 253us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:32.099071 (+ 415us) server_negotiation.cc:300] Negotiation successful
1023 20:17:32.099174 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20251023 20:17:32.100191 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:32.100587 32172 catalog_manager.cc:1357] Generated new cluster ID: ef9c8794aaa7498d86e1694e47788fb7
I20251023 20:17:32.100636 32172 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:32.108397 32172 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:32.109208 32172 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:32.109400 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:32.102000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48075 (local address 127.9.117.254:37657)
1023 20:17:32.102036 (+ 36us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:32.102038 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:32.102167 (+ 129us) server_negotiation.cc:408] Connection header received
1023 20:17:32.102252 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:32.102253 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:32.102262 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:32.102324 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:32.103184 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:32.103918 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:32.104779 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:32.104941 (+ 162us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:32.106565 (+ 1624us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:32.106571 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:32.106571 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:32.106578 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:32.108199 (+ 1621us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:32.108641 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:32.108642 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:32.108643 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:32.108669 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:32.108892 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:32.108893 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:32.108893 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:32.108989 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:32.109092 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:32.109234 (+ 142us) server_negotiation.cc:300] Negotiation successful
1023 20:17:32.109329 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20251023 20:17:32.113122 32172 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4c59668051c14949a0b8b90b3cd6b122: Generated new TSK 0
I20251023 20:17:32.113353 32172 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:32.120273 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 32090
W20251023 20:17:32.125412 32072 connection.cc:537] client connection to 127.9.117.254:37657 recv error: Network error: failed to read from TLS socket (remote: 127.9.117.254:37657): Connection reset by peer (error 104)
2025-10-23T20:17:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:32 dist-test-slave-x62p krb5kdc[32189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:32 dist-test-slave-x62p krb5kdc[32189](info): set up 2 sockets
Oct 23 20:17:32 dist-test-slave-x62p krb5kdc[32189](info): commencing operation
krb5kdc: starting...
W20251023 20:17:34.150681 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:34 dist-test-slave-x62p krb5kdc[32189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:34.297534 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40117
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:34.380177 32205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:34.380370 32205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:34.380396 32205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:34.381923 32205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:34.381969 32205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:34.381984 32205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:34.381997 32205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:34.382009 32205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:34.383612 32205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:40117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:34.383801 32205 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:34.383945 32205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:34.386503 32210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:34.386529 32211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:34.386693 32213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:34.386673 32205 server_base.cc:1047] running on GCE node
I20251023 20:17:34.386945 32205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:34.387156 32205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:34.388286 32205 hybrid_clock.cc:648] HybridClock initialized: now 1761250654388273 us; error 30 us; skew 500 ppm
Oct 23 20:17:34 dist-test-slave-x62p krb5kdc[32189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:34.390810 32205 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:17:34.391386 32205 webserver.cc:492] Webserver started at http://127.9.117.254:33015/ using document root <none> and password file <none>
I20251023 20:17:34.391572 32205 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:34.391614 32205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:34.391739 32205 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:34.392699 32205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4"
format_stamp: "Formatted at 2025-10-23 20:17:34 on dist-test-slave-x62p"
server_key: "b754b55aa070eb9cf530fbbf6ef1f205"
server_key_iv: "af3d4b01b6fbd52fee5b04ead04287bf"
server_key_version: "encryptionkey@0"
I20251023 20:17:34.393057 32205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4"
format_stamp: "Formatted at 2025-10-23 20:17:34 on dist-test-slave-x62p"
server_key: "b754b55aa070eb9cf530fbbf6ef1f205"
server_key_iv: "af3d4b01b6fbd52fee5b04ead04287bf"
server_key_version: "encryptionkey@0"
I20251023 20:17:34.394309 32205 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:34.395040 32220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:34.395242 32205 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:34.395325 32205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4"
format_stamp: "Formatted at 2025-10-23 20:17:34 on dist-test-slave-x62p"
server_key: "b754b55aa070eb9cf530fbbf6ef1f205"
server_key_iv: "af3d4b01b6fbd52fee5b04ead04287bf"
server_key_version: "encryptionkey@0"
I20251023 20:17:34.395393 32205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:34.403424 32205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:34.405547 32205 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:34.405702 32205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:34.410063 32205 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:40117
I20251023 20:17:34.410112 32272 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:40117 every 8 connection(s)
I20251023 20:17:34.410429 32205 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:34.411046 32273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:34.412376 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 32205
I20251023 20:17:34.412518 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:34.412715 9687 external_mini_cluster.cc:1468] Setting key 9d7e9f708a5ac1b6df1ad19544dbd82f
I20251023 20:17:34.414700 32273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4: Bootstrap starting.
I20251023 20:17:34.415289 32273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:34.415578 32273 log.cc:826] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:34.416333 32273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4: No bootstrap required, opened a new log
I20251023 20:17:34.417536 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:34.413293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42295 (local address 127.9.117.254:40117)
1023 20:17:34.413450 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:34.413459 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:34.413482 (+ 23us) server_negotiation.cc:408] Connection header received
1023 20:17:34.413864 (+ 382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:34.413870 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:34.413912 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:34.414035 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:34.414517 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:34.415276 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:34.416064 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:34.416251 (+ 187us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:34.416888 (+ 637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:34.416895 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:34.416896 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:34.417063 (+ 167us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:34.417082 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:34.417089 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:34.417165 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:34.417334 (+ 169us) server_negotiation.cc:300] Negotiation successful
1023 20:17:34.417366 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20251023 20:17:34.417927 32273 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40117 } }
I20251023 20:17:34.418054 32273 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:34.418088 32273 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bebcbcfe34a4f6aab1c64ca880dbfc4, State: Initialized, Role: FOLLOWER
I20251023 20:17:34.418191 32273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40117 } }
I20251023 20:17:34.418256 32273 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:34.418279 32273 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:34.418306 32273 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:34.418460 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:34.418902 32273 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40117 } }
I20251023 20:17:34.419010 32273 leader_election.cc:304] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bebcbcfe34a4f6aab1c64ca880dbfc4; no voters:
I20251023 20:17:34.419159 32273 leader_election.cc:290] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:34.419209 32278 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:34.419323 32278 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [term 1 LEADER]: Becoming Leader. State: Replica: 1bebcbcfe34a4f6aab1c64ca880dbfc4, State: Running, Role: LEADER
I20251023 20:17:34.419400 32273 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:34.419399 32278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40117 } }
I20251023 20:17:34.419762 32280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40117 } } }
I20251023 20:17:34.419905 32280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:34.420156 32286 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:34.419754 32278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bebcbcfe34a4f6aab1c64ca880dbfc4. Latest consensus state: current_term: 1 leader_uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bebcbcfe34a4f6aab1c64ca880dbfc4" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 40117 } } }
I20251023 20:17:34.420384 32278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:34.420663 32286 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:34.422233 32286 catalog_manager.cc:1357] Generated new cluster ID: b5388020f6c443c29147cfaa933153d8
I20251023 20:17:34.422282 32286 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:34.425161 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:34.421484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58797 (local address 127.9.117.254:40117)
1023 20:17:34.421528 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:34.421531 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:34.421668 (+ 137us) server_negotiation.cc:408] Connection header received
1023 20:17:34.421732 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:34.421733 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:34.421744 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:34.421804 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:34.422188 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:34.422920 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:34.423705 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:34.423878 (+ 173us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:34.424782 (+ 904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:34.424785 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:34.424785 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:34.424791 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:34.424808 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:34.424812 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:34.424854 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:34.425071 (+ 217us) server_negotiation.cc:300] Negotiation successful
1023 20:17:34.425085 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251023 20:17:34.447924 32286 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:34.448818 32286 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:34.461561 32286 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1bebcbcfe34a4f6aab1c64ca880dbfc4: Generated new TSK 0
I20251023 20:17:34.461826 32286 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:34.464865 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 32205
W20251023 20:17:34.472060 32187 connection.cc:537] client connection to 127.9.117.254:40117 recv error: Network error: failed to read from TLS socket (remote: 127.9.117.254:40117): Connection reset by peer (error 104)
2025-10-23T20:17:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:34 dist-test-slave-x62p krb5kdc[32304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:34 dist-test-slave-x62p krb5kdc[32304](info): set up 2 sockets
Oct 23 20:17:34 dist-test-slave-x62p krb5kdc[32304](info): commencing operation
krb5kdc: starting...
W20251023 20:17:36.505908 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:36 dist-test-slave-x62p krb5kdc[32304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:36.668326 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:36069
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:36069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:36.751348 32320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:36.751559 32320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:36.751583 32320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:36.753165 32320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:36.753212 32320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:36.753227 32320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:36.753240 32320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:36.753252 32320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:36.754933 32320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:36069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:36069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:36.755172 32320 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:36.755376 32320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:36.758317 32326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:36.758389 32320 server_base.cc:1047] running on GCE node
W20251023 20:17:36.758431 32328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:36.758325 32325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:36.758792 32320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:36.759128 32320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:36.760277 32320 hybrid_clock.cc:648] HybridClock initialized: now 1761250656760260 us; error 38 us; skew 500 ppm
Oct 23 20:17:36 dist-test-slave-x62p krb5kdc[32304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250656, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:36.762799 32320 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:17:36.763355 32320 webserver.cc:492] Webserver started at http://127.9.117.254:37411/ using document root <none> and password file <none>
I20251023 20:17:36.763547 32320 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:36.763586 32320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:36.763691 32320 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:36.764652 32320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "85cb593f291148619acf2db0928ab22a"
format_stamp: "Formatted at 2025-10-23 20:17:36 on dist-test-slave-x62p"
server_key: "da39694e17975c67f6dcddca789b380c"
server_key_iv: "16355b2d95991c606f4dee5b26cd714e"
server_key_version: "encryptionkey@0"
I20251023 20:17:36.765010 32320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "85cb593f291148619acf2db0928ab22a"
format_stamp: "Formatted at 2025-10-23 20:17:36 on dist-test-slave-x62p"
server_key: "da39694e17975c67f6dcddca789b380c"
server_key_iv: "16355b2d95991c606f4dee5b26cd714e"
server_key_version: "encryptionkey@0"
I20251023 20:17:36.766305 32320 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:36.767230 32335 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:36.767409 32320 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:36.767480 32320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "85cb593f291148619acf2db0928ab22a"
format_stamp: "Formatted at 2025-10-23 20:17:36 on dist-test-slave-x62p"
server_key: "da39694e17975c67f6dcddca789b380c"
server_key_iv: "16355b2d95991c606f4dee5b26cd714e"
server_key_version: "encryptionkey@0"
I20251023 20:17:36.767555 32320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:36.771087 32320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:36.777484 32320 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:36.777606 32320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:36.781889 32320 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:36069
I20251023 20:17:36.781940 32387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:36069 every 8 connection(s)
I20251023 20:17:36.782253 32320 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:36.782913 32388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:36.783032 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 32320
I20251023 20:17:36.783154 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:36.783334 9687 external_mini_cluster.cc:1468] Setting key f01343643dbd764ddcf6f7e052b11226
I20251023 20:17:36.786160 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:36.783843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55865 (local address 127.9.117.254:36069)
1023 20:17:36.783995 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:36.784007 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:36.784027 (+ 20us) server_negotiation.cc:408] Connection header received
1023 20:17:36.784894 (+ 867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:36.784902 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:36.784933 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:36.785050 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:36.785759 (+ 709us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:36.785762 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:36.785762 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:36.785766 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:36.785790 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:36.785797 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:36.785904 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:36.785944 (+ 40us) server_negotiation.cc:300] Negotiation successful
1023 20:17:36.785977 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20251023 20:17:36.786557 32388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a: Bootstrap starting.
I20251023 20:17:36.786955 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:36.787173 32388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:36.787426 32388 log.cc:826] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:36.788139 32388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a: No bootstrap required, opened a new log
I20251023 20:17:36.789489 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:36.788413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49319 (local address 127.9.117.254:36069)
1023 20:17:36.788482 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:36.788484 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:36.788540 (+ 56us) server_negotiation.cc:408] Connection header received
1023 20:17:36.788620 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:36.788621 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:36.788625 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:36.788671 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:36.789274 (+ 603us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:36.789275 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:36.789275 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:36.789279 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:36.789293 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:36.789296 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:36.789341 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:36.789401 (+ 60us) server_negotiation.cc:300] Negotiation successful
1023 20:17:36.789412 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251023 20:17:36.789871 32388 raft_consensus.cc:359] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cb593f291148619acf2db0928ab22a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36069 } }
I20251023 20:17:36.790002 32388 raft_consensus.cc:385] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:36.790032 32388 raft_consensus.cc:740] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cb593f291148619acf2db0928ab22a, State: Initialized, Role: FOLLOWER
I20251023 20:17:36.790138 32388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cb593f291148619acf2db0928ab22a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36069 } }
I20251023 20:17:36.790195 32388 raft_consensus.cc:399] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:36.790237 32388 raft_consensus.cc:493] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:36.790294 32388 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:36.790908 32388 raft_consensus.cc:515] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cb593f291148619acf2db0928ab22a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36069 } }
I20251023 20:17:36.791013 32388 leader_election.cc:304] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85cb593f291148619acf2db0928ab22a; no voters:
I20251023 20:17:36.791160 32388 leader_election.cc:290] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:36.791211 32398 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:36.791368 32398 raft_consensus.cc:697] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [term 1 LEADER]: Becoming Leader. State: Replica: 85cb593f291148619acf2db0928ab22a, State: Running, Role: LEADER
I20251023 20:17:36.791404 32388 sys_catalog.cc:565] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:36.791513 32398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cb593f291148619acf2db0928ab22a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36069 } }
I20251023 20:17:36.791862 32399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85cb593f291148619acf2db0928ab22a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cb593f291148619acf2db0928ab22a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36069 } } }
I20251023 20:17:36.791893 32400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85cb593f291148619acf2db0928ab22a. Latest consensus state: current_term: 1 leader_uuid: "85cb593f291148619acf2db0928ab22a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cb593f291148619acf2db0928ab22a" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 36069 } } }
I20251023 20:17:36.792090 32400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:36.792011 32399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:36.792410 32405 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:36.792951 32405 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:36.794530 32405 catalog_manager.cc:1357] Generated new cluster ID: ea1f120a7fd644a68f848004fa4c687c
I20251023 20:17:36.794584 32405 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:36.800571 32405 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:36.800773 32405 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:36.805086 32405 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 85cb593f291148619acf2db0928ab22a: Generated new TSK 0
I20251023 20:17:36.805282 32405 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:36.810706 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 32320
2025-10-23T20:17:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:36 dist-test-slave-x62p krb5kdc[32419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:36 dist-test-slave-x62p krb5kdc[32419](info): set up 2 sockets
Oct 23 20:17:36 dist-test-slave-x62p krb5kdc[32419](info): commencing operation
krb5kdc: starting...
W20251023 20:17:38.877621 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:38 dist-test-slave-x62p krb5kdc[32419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:39.028102 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33527
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:33527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:39.115087 32435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:39.115307 32435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:39.115334 32435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:39.116991 32435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:39.117051 32435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:39.117067 32435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:39.117080 32435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:39.117092 32435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:39.118909 32435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:33527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:33527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:39.119171 32435 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:39.119371 32435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:39.122079 32440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:39.122097 32443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:39.122104 32441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:39.122473 32435 server_base.cc:1047] running on GCE node
I20251023 20:17:39.122702 32435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:39.122958 32435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:39.124118 32435 hybrid_clock.cc:648] HybridClock initialized: now 1761250659124088 us; error 47 us; skew 500 ppm
Oct 23 20:17:39 dist-test-slave-x62p krb5kdc[32419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250659, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:39.127000 32435 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:17:39.127626 32435 webserver.cc:492] Webserver started at http://127.9.117.254:40553/ using document root <none> and password file <none>
I20251023 20:17:39.127827 32435 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:39.127869 32435 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:39.127988 32435 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:39.129078 32435 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "55d1720976834fc98926bf0f30cf0213"
format_stamp: "Formatted at 2025-10-23 20:17:39 on dist-test-slave-x62p"
server_key: "051a8d842daa73d0884935fe6b961e91"
server_key_iv: "777d2aafe12b036931f3a1047facd1b0"
server_key_version: "encryptionkey@0"
I20251023 20:17:39.129397 32435 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "55d1720976834fc98926bf0f30cf0213"
format_stamp: "Formatted at 2025-10-23 20:17:39 on dist-test-slave-x62p"
server_key: "051a8d842daa73d0884935fe6b961e91"
server_key_iv: "777d2aafe12b036931f3a1047facd1b0"
server_key_version: "encryptionkey@0"
I20251023 20:17:39.130702 32435 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:17:39.131392 32450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:39.131614 32435 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251023 20:17:39.131691 32435 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "55d1720976834fc98926bf0f30cf0213"
format_stamp: "Formatted at 2025-10-23 20:17:39 on dist-test-slave-x62p"
server_key: "051a8d842daa73d0884935fe6b961e91"
server_key_iv: "777d2aafe12b036931f3a1047facd1b0"
server_key_version: "encryptionkey@0"
I20251023 20:17:39.131765 32435 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:39.141705 32435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:39.143937 32435 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:39.144102 32435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:39.148680 32435 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:33527
I20251023 20:17:39.148828 32502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:33527 every 8 connection(s)
I20251023 20:17:39.149142 32435 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:39.149778 32503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:39.152503 32503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213: Bootstrap starting.
I20251023 20:17:39.153224 32503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:39.153326 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 32435
I20251023 20:17:39.153479 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:39.153571 32503 log.cc:826] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:39.153685 9687 external_mini_cluster.cc:1468] Setting key 2f30a7ae078059faa2631fd441bc34bb
I20251023 20:17:39.154476 32503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213: No bootstrap required, opened a new log
I20251023 20:17:39.156226 32503 raft_consensus.cc:359] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1720976834fc98926bf0f30cf0213" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33527 } }
I20251023 20:17:39.156368 32503 raft_consensus.cc:385] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:39.156402 32503 raft_consensus.cc:740] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d1720976834fc98926bf0f30cf0213, State: Initialized, Role: FOLLOWER
I20251023 20:17:39.156522 32503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1720976834fc98926bf0f30cf0213" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33527 } }
I20251023 20:17:39.156595 32503 raft_consensus.cc:399] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:39.156632 32503 raft_consensus.cc:493] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:39.156682 32503 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:39.157452 32503 raft_consensus.cc:515] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1720976834fc98926bf0f30cf0213" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33527 } }
I20251023 20:17:39.157572 32503 leader_election.cc:304] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55d1720976834fc98926bf0f30cf0213; no voters:
I20251023 20:17:39.157733 32503 leader_election.cc:290] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:39.157825 32508 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:39.157980 32508 raft_consensus.cc:697] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [term 1 LEADER]: Becoming Leader. State: Replica: 55d1720976834fc98926bf0f30cf0213, State: Running, Role: LEADER
I20251023 20:17:39.158102 32508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1720976834fc98926bf0f30cf0213" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33527 } }
I20251023 20:17:39.158399 32503 sys_catalog.cc:565] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:39.158500 32510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55d1720976834fc98926bf0f30cf0213. Latest consensus state: current_term: 1 leader_uuid: "55d1720976834fc98926bf0f30cf0213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1720976834fc98926bf0f30cf0213" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33527 } } }
I20251023 20:17:39.158502 32509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55d1720976834fc98926bf0f30cf0213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1720976834fc98926bf0f30cf0213" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 33527 } } }
I20251023 20:17:39.158623 32510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:39.158625 32509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:39.158875 32512 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:39.159526 32512 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 23 20:17:39 dist-test-slave-x62p krb5kdc[32419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:39.161132 32512 catalog_manager.cc:1357] Generated new cluster ID: eea0305ad8944f95b813c66f204502f1
I20251023 20:17:39.161192 32512 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:39.164760 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:39.154246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48511 (local address 127.9.117.254:33527)
1023 20:17:39.154436 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:39.154456 (+ 20us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:39.154483 (+ 27us) server_negotiation.cc:408] Connection header received
1023 20:17:39.154883 (+ 400us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:39.154890 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:39.154936 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:39.155055 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:39.156073 (+ 1018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:39.157017 (+ 944us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:39.158078 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:39.158355 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:39.161146 (+ 2791us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:39.161173 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:39.161174 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:39.161199 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:39.163266 (+ 2067us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:39.163682 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:39.163684 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:39.163685 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:39.163742 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:39.163910 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:39.163910 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:39.163911 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:39.164156 (+ 245us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:39.164275 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:39.164431 (+ 156us) server_negotiation.cc:300] Negotiation successful
1023 20:17:39.164548 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20251023 20:17:39.165669 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:39.174211 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:39.167331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45773 (local address 127.9.117.254:33527)
1023 20:17:39.167375 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:39.167376 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:39.167479 (+ 103us) server_negotiation.cc:408] Connection header received
1023 20:17:39.167541 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:39.167541 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:39.167551 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:39.167607 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:39.168428 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:39.169139 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:39.169954 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:39.170126 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:39.171350 (+ 1224us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:39.171355 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:39.171355 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:39.171362 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:39.173109 (+ 1747us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:39.173499 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:39.173500 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:39.173500 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:39.173525 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:39.173684 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:39.173685 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:39.173685 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:39.173800 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:39.173891 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:39.174016 (+ 125us) server_negotiation.cc:300] Negotiation successful
1023 20:17:39.174112 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20251023 20:17:39.175663 32512 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:39.176555 32512 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:39.185122 32512 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 55d1720976834fc98926bf0f30cf0213: Generated new TSK 0
I20251023 20:17:39.185344 32512 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:39.188313 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 32435
2025-10-23T20:17:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:39 dist-test-slave-x62p krb5kdc[32534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:39 dist-test-slave-x62p krb5kdc[32534](info): set up 2 sockets
Oct 23 20:17:39 dist-test-slave-x62p krb5kdc[32534](info): commencing operation
krb5kdc: starting...
W20251023 20:17:41.233314 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.026s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:41 dist-test-slave-x62p krb5kdc[32534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:41.401254 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:42753
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:42753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:41.489878 32550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:41.490104 32550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:41.490144 32550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:41.491760 32550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:41.491823 32550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:41.491849 32550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:41.491868 32550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:41.491890 32550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:41.493721 32550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:42753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:42753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:41.494001 32550 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:41.494208 32550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:41.497143 32555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:41.497267 32556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:41.497143 32558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:41.497390 32550 server_base.cc:1047] running on GCE node
I20251023 20:17:41.497645 32550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:41.497903 32550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:41.499079 32550 hybrid_clock.cc:648] HybridClock initialized: now 1761250661499063 us; error 31 us; skew 500 ppm
Oct 23 20:17:41 dist-test-slave-x62p krb5kdc[32534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250661, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:41.502274 32550 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:17:41.503106 32550 webserver.cc:492] Webserver started at http://127.9.117.254:43035/ using document root <none> and password file <none>
I20251023 20:17:41.503330 32550 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:41.503376 32550 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:41.503633 32550 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:41.504850 32550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "700e3203b7864fafa40cb9cb15945f0f"
format_stamp: "Formatted at 2025-10-23 20:17:41 on dist-test-slave-x62p"
server_key: "85c88aad7ce449a59cb0d3c227d718a8"
server_key_iv: "cb2a1d5e21916b73dbd00503f899e6eb"
server_key_version: "encryptionkey@0"
I20251023 20:17:41.505192 32550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "700e3203b7864fafa40cb9cb15945f0f"
format_stamp: "Formatted at 2025-10-23 20:17:41 on dist-test-slave-x62p"
server_key: "85c88aad7ce449a59cb0d3c227d718a8"
server_key_iv: "cb2a1d5e21916b73dbd00503f899e6eb"
server_key_version: "encryptionkey@0"
I20251023 20:17:41.506685 32550 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:17:41.507635 32565 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:41.507898 32550 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:41.507993 32550 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "700e3203b7864fafa40cb9cb15945f0f"
format_stamp: "Formatted at 2025-10-23 20:17:41 on dist-test-slave-x62p"
server_key: "85c88aad7ce449a59cb0d3c227d718a8"
server_key_iv: "cb2a1d5e21916b73dbd00503f899e6eb"
server_key_version: "encryptionkey@0"
I20251023 20:17:41.508073 32550 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:41.521679 32550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:41.524070 32550 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:41.524222 32550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:41.528654 32550 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:42753
I20251023 20:17:41.528687 32617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:42753 every 8 connection(s)
I20251023 20:17:41.529055 32550 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:41.529743 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:41.532420 32618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f: Bootstrap starting.
I20251023 20:17:41.533165 32618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:41.533429 32618 log.cc:826] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:41.534361 32618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f: No bootstrap required, opened a new log
I20251023 20:17:41.535625 32618 raft_consensus.cc:359] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700e3203b7864fafa40cb9cb15945f0f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42753 } }
I20251023 20:17:41.535756 32618 raft_consensus.cc:385] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:41.535789 32618 raft_consensus.cc:740] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700e3203b7864fafa40cb9cb15945f0f, State: Initialized, Role: FOLLOWER
I20251023 20:17:41.535910 32618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700e3203b7864fafa40cb9cb15945f0f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42753 } }
I20251023 20:17:41.535987 32618 raft_consensus.cc:399] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:41.536031 32618 raft_consensus.cc:493] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:41.536079 32618 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:41.536665 32618 raft_consensus.cc:515] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700e3203b7864fafa40cb9cb15945f0f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42753 } }
I20251023 20:17:41.536813 32618 leader_election.cc:304] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700e3203b7864fafa40cb9cb15945f0f; no voters:
I20251023 20:17:41.536963 32618 leader_election.cc:290] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:41.537243 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 32550
I20251023 20:17:41.537200 32618 sys_catalog.cc:565] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:41.537379 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:41.537626 9687 external_mini_cluster.cc:1468] Setting key afe2a08756ce638fb69af9e80dfd3282
I20251023 20:17:41.537766 32621 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:41.537954 32621 raft_consensus.cc:697] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [term 1 LEADER]: Becoming Leader. State: Replica: 700e3203b7864fafa40cb9cb15945f0f, State: Running, Role: LEADER
I20251023 20:17:41.538065 32621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700e3203b7864fafa40cb9cb15945f0f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42753 } }
I20251023 20:17:41.538527 32623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700e3203b7864fafa40cb9cb15945f0f. Latest consensus state: current_term: 1 leader_uuid: "700e3203b7864fafa40cb9cb15945f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700e3203b7864fafa40cb9cb15945f0f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42753 } } }
I20251023 20:17:41.538610 32623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:41.539104 32622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700e3203b7864fafa40cb9cb15945f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700e3203b7864fafa40cb9cb15945f0f" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 42753 } } }
I20251023 20:17:41.539450 32622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:41.539570 32632 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:41.540107 32632 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:41.541455 32632 catalog_manager.cc:1357] Generated new cluster ID: dfa4015d6a4542dc92581f96906d2a00
I20251023 20:17:41.541504 32632 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:41.546656 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:41.538210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36657 (local address 127.9.117.254:42753)
1023 20:17:41.538420 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:41.538436 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:41.538998 (+ 562us) server_negotiation.cc:408] Connection header received
1023 20:17:41.539137 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:41.539144 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:41.539211 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:41.539343 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:41.540026 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:41.541327 (+ 1301us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:41.542170 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:41.542415 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:41.545899 (+ 3484us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:41.545908 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:41.545908 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:41.546162 (+ 254us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:41.546188 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:41.546197 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:41.546281 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:41.546410 (+ 129us) server_negotiation.cc:300] Negotiation successful
1023 20:17:41.546465 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20251023 20:17:41.547653 32632 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:41.547853 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:41.548455 32632 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:41.553049 32632 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 700e3203b7864fafa40cb9cb15945f0f: Generated new TSK 0
I20251023 20:17:41.553267 32632 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:41.556553 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:41.552705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50129 (local address 127.9.117.254:42753)
1023 20:17:41.552813 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:41.552816 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:41.553015 (+ 199us) server_negotiation.cc:408] Connection header received
1023 20:17:41.553160 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:41.553161 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:41.553175 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:41.553251 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:41.553721 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:41.554573 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:41.555247 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:41.555446 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:41.556232 (+ 786us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:41.556236 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:41.556236 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:41.556242 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:41.556260 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:41.556264 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:41.556332 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:41.556426 (+ 94us) server_negotiation.cc:300] Negotiation successful
1023 20:17:41.556461 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20251023 20:17:41.559111 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 32550
2025-10-23T20:17:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:41 dist-test-slave-x62p krb5kdc[32649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:41 dist-test-slave-x62p krb5kdc[32649](info): set up 2 sockets
Oct 23 20:17:41 dist-test-slave-x62p krb5kdc[32649](info): commencing operation
krb5kdc: starting...
W20251023 20:17:43.615446 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:43 dist-test-slave-x62p krb5kdc[32649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:43.777130 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:44957
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:40937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:44957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:43.865041 32665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:43.865257 32665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:43.865298 32665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:43.866915 32665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:43.866972 32665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:43.866999 32665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:43.867017 32665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:43.867035 32665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:43.868825 32665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:40937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:44957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:44957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:43.869112 32665 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:43.869305 32665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:43.874549 32673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:43.874640 32670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:43.874629 32671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:43.874682 32665 server_base.cc:1047] running on GCE node
I20251023 20:17:43.875068 32665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:43.875319 32665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:43.879729 32665 hybrid_clock.cc:648] HybridClock initialized: now 1761250663879671 us; error 58 us; skew 500 ppm
Oct 23 20:17:43 dist-test-slave-x62p krb5kdc[32649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250663, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:43.882864 32665 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:17:43.885694 32665 webserver.cc:492] Webserver started at http://127.9.117.254:45637/ using document root <none> and password file <none>
I20251023 20:17:43.885936 32665 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:43.885984 32665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:43.886222 32665 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:43.887909 32665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac"
format_stamp: "Formatted at 2025-10-23 20:17:43 on dist-test-slave-x62p"
server_key: "67f44c7e5a30b22bf54350a453fa147e"
server_key_iv: "6d4730be08632b2cdcf9600541adb2fb"
server_key_version: "encryptionkey@0"
I20251023 20:17:43.888535 32665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac"
format_stamp: "Formatted at 2025-10-23 20:17:43 on dist-test-slave-x62p"
server_key: "67f44c7e5a30b22bf54350a453fa147e"
server_key_iv: "6d4730be08632b2cdcf9600541adb2fb"
server_key_version: "encryptionkey@0"
I20251023 20:17:43.894327 32665 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20251023 20:17:43.895648 32680 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:43.895992 32665 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:43.896106 32665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac"
format_stamp: "Formatted at 2025-10-23 20:17:43 on dist-test-slave-x62p"
server_key: "67f44c7e5a30b22bf54350a453fa147e"
server_key_iv: "6d4730be08632b2cdcf9600541adb2fb"
server_key_version: "encryptionkey@0"
I20251023 20:17:43.896198 32665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:43.900563 32665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:43.903517 32665 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:43.903688 32665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:43.917196 32665 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:44957
I20251023 20:17:43.917276 32732 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:44957 every 8 connection(s)
I20251023 20:17:43.917632 32665 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:43.918435 32733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:43.921237 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac: Bootstrap starting.
I20251023 20:17:43.921918 32733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:43.922178 32733 log.cc:826] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:43.923053 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac: No bootstrap required, opened a new log
I20251023 20:17:43.924582 32733 raft_consensus.cc:359] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 44957 } }
I20251023 20:17:43.924813 32733 raft_consensus.cc:385] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:17:43.924808 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 32665
I20251023 20:17:43.924852 32733 raft_consensus.cc:740] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12c4e1e33bb4392b4c2e62699b3a5ac, State: Initialized, Role: FOLLOWER
I20251023 20:17:43.924922 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:43.925005 32733 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 44957 } }
I20251023 20:17:43.925076 32733 raft_consensus.cc:399] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251023 20:17:43.925098 9687 external_mini_cluster.cc:1468] Setting key 4dde6654701a9801df697a8e79d03e54
I20251023 20:17:43.925107 32733 raft_consensus.cc:493] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251023 20:17:43.925153 32733 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:43.926172 32733 raft_consensus.cc:515] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 44957 } }
I20251023 20:17:43.926293 32733 leader_election.cc:304] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12c4e1e33bb4392b4c2e62699b3a5ac; no voters:
I20251023 20:17:43.926430 32733 leader_election.cc:290] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20251023 20:17:43.926492 32738 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:43.926601 32738 raft_consensus.cc:697] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [term 1 LEADER]: Becoming Leader. State: Replica: a12c4e1e33bb4392b4c2e62699b3a5ac, State: Running, Role: LEADER
I20251023 20:17:43.926693 32733 sys_catalog.cc:565] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:43.926674 32738 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 44957 } }
I20251023 20:17:43.927022 32738 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12c4e1e33bb4392b4c2e62699b3a5ac. Latest consensus state: current_term: 1 leader_uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 44957 } } }
I20251023 20:17:43.927505 32738 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:43.927026 32739 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12c4e1e33bb4392b4c2e62699b3a5ac" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 44957 } } }
I20251023 20:17:43.927876 32739 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:43.928288 32747 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:43.930648 32747 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:43.931454 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:43.925838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48811 (local address 127.9.117.254:44957)
1023 20:17:43.926838 (+ 1000us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:43.926853 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:43.926885 (+ 32us) server_negotiation.cc:408] Connection header received
1023 20:17:43.927048 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:43.927055 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:43.927401 (+ 346us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:43.928430 (+ 1029us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:43.930317 (+ 1887us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:43.930322 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:43.930323 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:43.930327 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:43.930348 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:43.930356 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:43.930478 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:43.930601 (+ 123us) server_negotiation.cc:300] Negotiation successful
1023 20:17:43.930866 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":242,"threads_started":1}
I20251023 20:17:43.932932 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:43.933046 32747 catalog_manager.cc:1357] Generated new cluster ID: 5762c9dbe30749868bb5ffb64f9c3f59
I20251023 20:17:43.933084 32747 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:43.936136 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:43.934705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51315 (local address 127.9.117.254:44957)
1023 20:17:43.934823 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:43.934826 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:43.934958 (+ 132us) server_negotiation.cc:408] Connection header received
1023 20:17:43.935052 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:43.935053 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:43.935059 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:43.935111 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:43.935892 (+ 781us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:43.935894 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:43.935894 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1023 20:17:43.935899 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:43.935917 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1023 20:17:43.935921 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:43.935985 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:43.936045 (+ 60us) server_negotiation.cc:300] Negotiation successful
1023 20:17:43.936060 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20251023 20:17:43.962906 32747 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:43.963248 32747 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:43.973647 32747 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a12c4e1e33bb4392b4c2e62699b3a5ac: Generated new TSK 0
I20251023 20:17:43.973879 32747 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:17:43.977152 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 32665
2025-10-23T20:17:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2418 ms)
[----------] 18 tests from AuthTokenIssuingTest (42570 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:44 dist-test-slave-x62p krb5kdc[32764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:44 dist-test-slave-x62p krb5kdc[32764](info): set up 2 sockets
Oct 23 20:17:44 dist-test-slave-x62p krb5kdc[32764](info): commencing operation
krb5kdc: starting...
W20251023 20:17:46.026026 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:46.189460 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37039
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37039,127.9.117.253:43507,127.9.117.252:35317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:46.277156 312 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251023 20:17:46.277421 312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:46.277451 312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:46.277503 312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:46.279227 312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:46.279297 312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:46.279321 312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:46.279346 312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:46.279366 312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:46.281317 312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37039,127.9.117.253:43507,127.9.117.252:35317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:46.281649 312 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:46.281862 312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:17:46.285013 312 server_base.cc:1047] running on GCE node
W20251023 20:17:46.284992 318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:46.284992 317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:46.284992 320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:46.285617 312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:46.285945 312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:46.287133 312 hybrid_clock.cc:648] HybridClock initialized: now 1761250666287121 us; error 48 us; skew 500 ppm
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:46.289942 312 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:17:46.290570 312 webserver.cc:492] Webserver started at http://127.9.117.254:45175/ using document root <none> and password file <none>
I20251023 20:17:46.290781 312 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:46.290822 312 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:46.290943 312 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:46.291929 312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "6627fce0896445b1ba5927cdd86236e2"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "79f3ea487d080a969b40049d1bfb803e"
server_key_iv: "2318d744fa7adb267ec9ab3e3e79a51a"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.292243 312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "6627fce0896445b1ba5927cdd86236e2"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "79f3ea487d080a969b40049d1bfb803e"
server_key_iv: "2318d744fa7adb267ec9ab3e3e79a51a"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.293568 312 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:17:46.294377 327 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:46.294696 312 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:46.294808 312 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "6627fce0896445b1ba5927cdd86236e2"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "79f3ea487d080a969b40049d1bfb803e"
server_key_iv: "2318d744fa7adb267ec9ab3e3e79a51a"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.294881 312 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:46.317201 312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:46.319315 312 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:46.319471 312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:46.324328 312 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:37039
I20251023 20:17:46.324419 379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:37039 every 8 connection(s)
I20251023 20:17:46.324779 312 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:46.325579 380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:46.327531 380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:46.334676 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 312
I20251023 20:17:46.334817 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:46.334988 9687 external_mini_cluster.cc:1468] Setting key 53d9c062572220bcb16a2eb731d1aa14
I20251023 20:17:46.337277 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.328258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37039 (local address 127.0.0.1:42938)
1023 20:17:46.328455 (+ 197us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.328485 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.328889 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.329179 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.329181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.329515 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.329789 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.329793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.330829 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.330830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.331196 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.331198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.331309 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.331938 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.331958 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.333535 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.335731 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.335739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.335742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.336013 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.336210 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.336211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.336211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.336287 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.336796 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.336803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.336875 (+ 72us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.336978 (+ 103us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.337117 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20251023 20:17:46.337402 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.328276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.9.117.254:37039)
1023 20:17:46.328443 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.328454 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.328575 (+ 121us) server_negotiation.cc:408] Connection header received
1023 20:17:46.329022 (+ 447us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.329024 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.329075 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.329157 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.329926 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.330737 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.331292 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.331469 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.333636 (+ 2167us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.333644 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.333645 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.333651 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.335624 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.336099 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.336101 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.336102 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.336149 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.336355 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.336356 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.336356 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.336660 (+ 304us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.336817 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.337018 (+ 201us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.337219 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20251023 20:17:46.338099 380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:46.338505 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.338382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:51952)
1023 20:17:46.338433 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.338478 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251023 20:17:46.338644 329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
I20251023 20:17:46.339108 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.339022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:51964)
1023 20:17:46.339065 (+ 43us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.339090 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251023 20:17:46.339211 380 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:43507: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.9.117.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.253@KRBTEST.COM" created.
I20251023 20:17:46.388449 380 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } attempt: 1
I20251023 20:17:46.389359 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.389169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:51974)
1023 20:17:46.389271 (+ 102us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.389319 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20251023 20:17:46.389990 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.389863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:51980)
1023 20:17:46.389919 (+ 56us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.389953 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20251023 20:17:46.390136 380 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:43507: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:46.399981 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.253:43507
--webserver_interface=127.9.117.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37039,127.9.117.253:43507,127.9.117.252:35317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:17:46.458334 380 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } attempt: 2
I20251023 20:17:46.458963 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.458781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:51994)
1023 20:17:46.458873 (+ 92us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.458924 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20251023 20:17:46.459396 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.459307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:52002)
1023 20:17:46.459348 (+ 41us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.459376 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20251023 20:17:46.459654 380 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:43507: Network error: Client connection negotiation failed: client connection to 127.9.117.253:43507: connect: Connection refused (error 111)
W20251023 20:17:46.495390 389 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251023 20:17:46.495625 389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:46.495642 389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:46.495676 389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:46.497471 389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:46.497529 389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:46.497543 389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:46.497556 389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:46.497568 389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:46.499507 389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37039,127.9.117.253:43507,127.9.117.252:35317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.253:43507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.117.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:46.499892 389 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:46.500118 389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:46.502868 395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:46.502894 394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:46.502995 397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:46.503108 389 server_base.cc:1047] running on GCE node
I20251023 20:17:46.503314 389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:46.503543 389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:46.504683 389 hybrid_clock.cc:648] HybridClock initialized: now 1761250666504663 us; error 35 us; skew 500 ppm
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:46.507295 389 init.cc:377] Logged in from keytab as kudu/127.9.117.253@KRBTEST.COM (short username kudu)
I20251023 20:17:46.507889 389 webserver.cc:492] Webserver started at http://127.9.117.253:32979/ using document root <none> and password file <none>
I20251023 20:17:46.508075 389 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:46.508118 389 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:46.508236 389 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:46.509289 389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/data/instance:
uuid: "873e996872814db6aa71565842291c3d"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "89acff07f0f1bd5752f16b2759f061c0"
server_key_iv: "1fc56b4f9fa6fbecacecf8fc2481e48a"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.509636 389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/wal/instance:
uuid: "873e996872814db6aa71565842291c3d"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "89acff07f0f1bd5752f16b2759f061c0"
server_key_iv: "1fc56b4f9fa6fbecacecf8fc2481e48a"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.511034 389 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:17:46.511895 404 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:46.512156 389 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:46.512223 389 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/wal
uuid: "873e996872814db6aa71565842291c3d"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "89acff07f0f1bd5752f16b2759f061c0"
server_key_iv: "1fc56b4f9fa6fbecacecf8fc2481e48a"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.512307 389 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:46.524629 389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:46.527200 389 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:46.527360 389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:46.531724 389 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.253:43507
I20251023 20:17:46.531924 456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.253:43507 every 8 connection(s)
I20251023 20:17:46.532126 389 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
I20251023 20:17:46.532855 457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:46.534821 457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:46.535821 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 389
I20251023 20:17:46.536026 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-1/wal/instance
I20251023 20:17:46.536209 9687 external_mini_cluster.cc:1468] Setting key a386d52ddadb977d78db410d73da4bea
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.117.252@KRBTEST.COM; defaulting to no policy
I20251023 20:17:46.544677 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.535588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.9.117.254:37039)
1023 20:17:46.535664 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.535667 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.535808 (+ 141us) server_negotiation.cc:408] Connection header received
1023 20:17:46.536231 (+ 423us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.536233 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.536251 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.536339 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.537423 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.538165 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.538803 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.538991 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.541346 (+ 2355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.541351 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.541352 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.541360 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.542892 (+ 1532us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.543467 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.543468 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.543468 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.543495 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.543880 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.543882 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.543882 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.544079 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.544149 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.544498 (+ 349us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.544601 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251023 20:17:46.544836 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.535504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37039 (local address 127.0.0.1:42954)
1023 20:17:46.535683 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.535716 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.536125 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.536399 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.536403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.536933 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.537263 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.537267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.538291 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.538292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.538699 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.538701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.538859 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.539475 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.539499 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.541204 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.543011 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.543024 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.543028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.543363 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.543648 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.543650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.543650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.543769 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.544174 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.544180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.544335 (+ 155us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.544441 (+ 106us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.544575 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20251023 20:17:46.545548 457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.252@KRBTEST.COM" created.
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
I20251023 20:17:46.550956 380 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } attempt: 3
I20251023 20:17:46.553660 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.545909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:52010)
1023 20:17:46.545979 (+ 70us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.545992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.546040 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.546426 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.546427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.546603 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.546752 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.546754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.547789 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.547790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.548150 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.548151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.548221 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.548853 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.548858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.550151 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.552651 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.552653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.552653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.552866 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.553060 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.553061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.553061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.553090 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.553320 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.553320 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.553358 (+ 38us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.553434 (+ 76us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.553541 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20251023 20:17:46.554122 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.546053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.9.117.253:43507)
1023 20:17:46.546256 (+ 203us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.546263 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.546289 (+ 26us) server_negotiation.cc:408] Connection header received
1023 20:17:46.546309 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.546311 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.546323 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.546401 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.546851 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.547649 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.548224 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.548443 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.550252 (+ 1809us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.550265 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.550265 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.550274 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.552505 (+ 2231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.552958 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.552963 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.552963 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.553000 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.553141 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.553142 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.553142 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.553262 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.553361 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.553912 (+ 551us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.554003 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":104,"threads_started":1}
I20251023 20:17:46.555042 457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:46.555547 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.555407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50586)
1023 20:17:46.555457 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.555512 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20251023 20:17:46.555830 405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
I20251023 20:17:46.556342 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.556251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50592)
1023 20:17:46.556287 (+ 36us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.556314 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20251023 20:17:46.556783 457 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:35317: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
I20251023 20:17:46.561233 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.551703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.9.117.253:43507)
1023 20:17:46.551908 (+ 205us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.551911 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.551925 (+ 14us) server_negotiation.cc:408] Connection header received
1023 20:17:46.551950 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.551951 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.551961 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.552045 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.552463 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.553342 (+ 879us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.554082 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.555004 (+ 922us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.556806 (+ 1802us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.556811 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.556812 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.556820 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.559382 (+ 2562us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.559949 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.559950 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.559951 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.559980 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.560371 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.560372 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.560373 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.560493 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.560598 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.561031 (+ 433us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.561132 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":129,"threads_started":1}
I20251023 20:17:46.561321 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.551310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:52024)
1023 20:17:46.551384 (+ 74us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.551398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.551456 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.552086 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.552087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.552227 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.552382 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.552383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.553426 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.553427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.553991 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.553995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.554073 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.554614 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.554618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.556531 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.559509 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.559511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.559511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.559775 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.560196 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.560197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.560197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.560230 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.560714 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.560715 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.560845 (+ 130us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.560941 (+ 96us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.561190 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20251023 20:17:46.562083 380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:46.562592 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.562439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50600)
1023 20:17:46.562521 (+ 82us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.562566 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20251023 20:17:46.563238 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.563069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50606)
1023 20:17:46.563165 (+ 96us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.563204 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20251023 20:17:46.563428 380 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:35317: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.252@KRBTEST.COM" created.
I20251023 20:17:46.594717 380 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } attempt: 1
I20251023 20:17:46.595398 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.595126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50616)
1023 20:17:46.595250 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.595298 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20251023 20:17:46.595911 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.595798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50630)
1023 20:17:46.595860 (+ 62us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.595887 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251023 20:17:46.596031 380 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:35317: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:46.602620 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.252:35317
--webserver_interface=127.9.117.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:34215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:37039,127.9.117.253:43507,127.9.117.252:35317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:17:46.606117 457 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } attempt: 1
I20251023 20:17:46.606788 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.606555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50638)
1023 20:17:46.606698 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.606744 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251023 20:17:46.607198 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.607103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50652)
1023 20:17:46.607149 (+ 46us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.607176 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20251023 20:17:46.607414 457 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:35317: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
I20251023 20:17:46.671240 380 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } attempt: 2
I20251023 20:17:46.671844 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.671680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50668)
1023 20:17:46.671751 (+ 71us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.671803 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20251023 20:17:46.672356 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.672175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50676)
1023 20:17:46.672287 (+ 112us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.672326 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20251023 20:17:46.672497 380 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:35317: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
I20251023 20:17:46.675627 457 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } attempt: 2
I20251023 20:17:46.676187 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.676019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50690)
1023 20:17:46.676093 (+ 74us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.676141 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20251023 20:17:46.676712 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.676601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50696)
1023 20:17:46.676653 (+ 52us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.676686 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20251023 20:17:46.676892 457 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:35317: Network error: Client connection negotiation failed: client connection to 127.9.117.252:35317: connect: Connection refused (error 111)
W20251023 20:17:46.700295 467 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251023 20:17:46.700570 467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:46.700590 467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:46.700616 467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:46.702771 467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:46.702845 467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:46.702862 467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:46.702876 467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:46.702889 467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:46.705281 467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:34215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:37039,127.9.117.253:43507,127.9.117.252:35317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.252:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.117.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:46.705581 467 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:46.705987 467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:46.709427 472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:46.709533 467 server_base.cc:1047] running on GCE node
W20251023 20:17:46.709410 473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:46.709519 475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:46.709941 467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:46.710218 467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:46.711421 467 hybrid_clock.cc:648] HybridClock initialized: now 1761250666711362 us; error 62 us; skew 500 ppm
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:46.714403 467 init.cc:377] Logged in from keytab as kudu/127.9.117.252@KRBTEST.COM (short username kudu)
I20251023 20:17:46.715456 467 webserver.cc:492] Webserver started at http://127.9.117.252:39407/ using document root <none> and password file <none>
I20251023 20:17:46.715667 467 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:46.715710 467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:46.715835 467 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:46.717139 467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/data/instance:
uuid: "f608ef5afef646ee81696d82b7f9abe7"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "2d6a61c17491cec850222ee76c9f09ce"
server_key_iv: "3770848201c29f9818ec6a96cca1c12b"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.717502 467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/wal/instance:
uuid: "f608ef5afef646ee81696d82b7f9abe7"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "2d6a61c17491cec850222ee76c9f09ce"
server_key_iv: "3770848201c29f9818ec6a96cca1c12b"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.718902 467 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:17:46.719761 482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:46.720031 467 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:46.720122 467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/wal
uuid: "f608ef5afef646ee81696d82b7f9abe7"
format_stamp: "Formatted at 2025-10-23 20:17:46 on dist-test-slave-x62p"
server_key: "2d6a61c17491cec850222ee76c9f09ce"
server_key_iv: "3770848201c29f9818ec6a96cca1c12b"
server_key_version: "encryptionkey@0"
I20251023 20:17:46.720203 467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:46.736440 467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:46.738881 467 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:46.739043 467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:46.744328 467 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.252:35317
I20251023 20:17:46.744403 534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.252:35317 every 8 connection(s)
I20251023 20:17:46.744845 467 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
I20251023 20:17:46.745554 535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:46.747536 535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:46.749301 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 467
I20251023 20:17:46.749429 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761250300132822-9687-0/minicluster-data/master-2/wal/instance
I20251023 20:17:46.749629 9687 external_mini_cluster.cc:1468] Setting key 07404beb5ebbe4e27a0804cd46b523e4
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:17:46.757915 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.748386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.9.117.254:37039)
1023 20:17:46.748531 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.748534 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.748752 (+ 218us) server_negotiation.cc:408] Connection header received
1023 20:17:46.749215 (+ 463us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.749216 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.749229 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.749298 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.750500 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.751063 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.751660 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.751841 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.754492 (+ 2651us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.754498 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.754498 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.754506 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.756242 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.756848 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.756850 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.756850 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.756881 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.757106 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.757107 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.757107 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.757218 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.757282 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.757599 (+ 317us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.757810 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20251023 20:17:46.758062 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.748311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:37039 (local address 127.0.0.1:42960)
1023 20:17:46.748583 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.748613 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.749072 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.749386 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.749393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.749889 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.750310 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.750315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.751184 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.751185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.751575 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.751578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.751707 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.752402 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.752424 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.754380 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.756354 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.756363 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.756367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.756715 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.756939 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.756940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.756941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.757037 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.757274 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.757280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.757448 (+ 168us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.757550 (+ 102us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.757843 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
I20251023 20:17:46.758955 535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:46.759920 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.750068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.9.117.252:35317)
1023 20:17:46.750243 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.750250 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.750265 (+ 15us) server_negotiation.cc:408] Connection header received
1023 20:17:46.750320 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.750323 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.750338 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.750409 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.751170 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.751834 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.752546 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.752819 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.755846 (+ 3027us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.755862 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.755863 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.755870 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.758473 (+ 2603us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.759051 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.759054 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.759055 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.759099 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.759269 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.759269 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.759270 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.759424 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.759590 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.759677 (+ 87us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.759799 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:46.767670 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.759332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:43507 (local address 127.0.0.1:52030)
1023 20:17:46.759382 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.759393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.759431 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.759620 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.759621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.759825 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.760014 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.760018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.761107 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.761108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.761610 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.761613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.761775 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.762745 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.762766 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.764562 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.766423 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.766424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.766425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.766617 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.766818 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.766818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.766819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.766854 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.767273 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.767274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.767341 (+ 67us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.767435 (+ 94us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.767582 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20251023 20:17:46.767732 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.759419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.9.117.253:43507)
1023 20:17:46.759493 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.759495 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.759503 (+ 8us) server_negotiation.cc:408] Connection header received
1023 20:17:46.759529 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.759530 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.759553 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.759609 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.760127 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.760934 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.761770 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.762080 (+ 310us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.764676 (+ 2596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.764681 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.764682 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.764690 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.766323 (+ 1633us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.766716 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.766717 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.766718 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.766749 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.766927 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.766928 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.766928 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.767080 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.767162 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.767502 (+ 340us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.767610 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251023 20:17:46.768168 457 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } attempt: 3
I20251023 20:17:46.768180 535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:46.770215 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.761839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.9.117.254:37039)
1023 20:17:46.762764 (+ 925us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.762768 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.762779 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:17:46.762803 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.762804 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.762814 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.762885 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.763561 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.764135 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.764688 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.764915 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.766988 (+ 2073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.766993 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.766993 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.767002 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.769074 (+ 2072us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.769438 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.769440 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.769440 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.769473 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.769630 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.769630 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.769631 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.769745 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.769897 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.769936 (+ 39us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.770030 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251023 20:17:46.771624 380 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } attempt: 3
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:17:46.775852 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.768631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50716)
1023 20:17:46.768661 (+ 30us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.768671 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.768711 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.768992 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.768993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.769147 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.769274 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.769276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.770290 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.770290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.770655 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.770656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.770726 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.771154 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.771159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.772651 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.775021 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.775022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.775023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.775154 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.775302 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.775303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.775304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.775331 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.775523 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.775524 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.775560 (+ 36us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.775627 (+ 67us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.775734 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20251023 20:17:46.775942 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.768654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.9.117.252:35317)
1023 20:17:46.768890 (+ 236us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.768892 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.768902 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:17:46.768917 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.768918 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.768925 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.768983 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.769344 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.770189 (+ 845us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.770727 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.771447 (+ 720us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.772782 (+ 1335us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.772786 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.772787 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.772793 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.774920 (+ 2127us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.775222 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.775223 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.775223 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.775245 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.775383 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.775384 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.775385 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.775469 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.775529 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.775650 (+ 121us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.775813 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":219,"thread_start_us":180,"threads_started":1}
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
Oct 23 20:17:46 dist-test-slave-x62p krb5kdc[32764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:17:46.777937 535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7: Bootstrap starting.
I20251023 20:17:46.778815 535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:46.779136 535 log.cc:826] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:46.780181 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.768466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50714)
1023 20:17:46.768522 (+ 56us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.768534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.768585 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.768982 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.768983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.769152 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.769343 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.769345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.770225 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.770226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.770697 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.770699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.770769 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.771530 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.771535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.773285 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.778901 (+ 5616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.778903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.778903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.779136 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.779436 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.779437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.779438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.779476 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.779825 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.779826 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.779880 (+ 54us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.779931 (+ 51us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.780059 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20251023 20:17:46.780604 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.768510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.9.117.252:35317)
1023 20:17:46.768563 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.768565 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.768574 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:17:46.768871 (+ 297us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.768873 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.768886 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.769426 (+ 540us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.769466 (+ 40us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.770126 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.770781 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.770973 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.773807 (+ 2834us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.773810 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.773811 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.773818 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.778784 (+ 4966us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.779303 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.779305 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.779305 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.779333 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.779556 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.779557 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.779557 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.779708 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.780379 (+ 671us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.780419 (+ 40us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.780517 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251023 20:17:46.781561 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.771886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:35317 (local address 127.0.0.1:50730)
1023 20:17:46.771949 (+ 63us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:46.771962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:46.772012 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:46.773185 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:46.773186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:46.773347 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:46.773551 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.773553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.775206 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.775208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:46.775826 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:46.775830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.775881 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.776320 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:46.776324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:46.777831 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:46.780143 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.780144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.780145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.780361 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.780915 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:46.780916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:46.780916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:46.780957 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:46.781222 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:46.781223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:46.781299 (+ 76us) client_negotiation.cc:769] Sending connection context
1023 20:17:46.781351 (+ 52us) client_negotiation.cc:240] Negotiation successful
1023 20:17:46.781480 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251023 20:17:46.781816 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.771529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.9.117.253:43507)
1023 20:17:46.771586 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.771588 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.772145 (+ 557us) server_negotiation.cc:408] Connection header received
1023 20:17:46.772246 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.772247 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.772258 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.772285 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.772976 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.773688 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.774588 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.774959 (+ 371us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.777249 (+ 2290us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.777254 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.777254 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.777263 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.779745 (+ 2482us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.780067 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.780068 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.780068 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.780097 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.780316 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.780317 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.780317 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.780429 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.781187 (+ 758us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.781235 (+ 48us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.781534 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251023 20:17:46.782012 535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7: No bootstrap required, opened a new log
I20251023 20:17:46.782099 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.772116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.9.117.252:35317)
1023 20:17:46.773046 (+ 930us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.773048 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.773061 (+ 13us) server_negotiation.cc:408] Connection header received
1023 20:17:46.773080 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.773081 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.773092 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.773639 (+ 547us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.773686 (+ 47us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.775084 (+ 1398us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.776066 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.777915 (+ 1849us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.777971 (+ 56us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.777975 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.777975 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.777980 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.780054 (+ 2074us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.780443 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.780444 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.780445 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.780582 (+ 137us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.781036 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.781037 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.781037 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.781145 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.781783 (+ 638us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.781825 (+ 42us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.781923 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":111,"threads_started":1}
I20251023 20:17:46.782831 457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d: Bootstrap starting.
I20251023 20:17:46.783516 457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:46.783974 457 log.cc:826] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:46.784314 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:46.784883 457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d: No bootstrap required, opened a new log
I20251023 20:17:46.784929 535 raft_consensus.cc:359] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } }
I20251023 20:17:46.785075 535 raft_consensus.cc:740] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f608ef5afef646ee81696d82b7f9abe7, State: Initialized, Role: FOLLOWER
I20251023 20:17:46.785138 535 consensus_queue.cc:260] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } }
I20251023 20:17:46.785440 535 sys_catalog.cc:565] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:46.785912 380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2: Bootstrap starting.
I20251023 20:17:46.786708 380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:46.785380 354 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6627fce0896445b1ba5927cdd86236e2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42964
I20251023 20:17:46.786823 457 raft_consensus.cc:359] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } }
I20251023 20:17:46.786976 457 raft_consensus.cc:740] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873e996872814db6aa71565842291c3d, State: Initialized, Role: FOLLOWER
I20251023 20:17:46.787029 380 log.cc:826] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:46.787041 457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } }
I20251023 20:17:46.787374 457 sys_catalog.cc:565] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:46.787395 544 sys_catalog.cc:455] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.787470 544 sys_catalog.cc:458] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [sys.catalog]: This master's current role is: FOLLOWER
W20251023 20:17:46.788569 555 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:46.788784 555 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:46.788899 380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2: No bootstrap required, opened a new log
I20251023 20:17:46.789429 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.789543 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:46.787922 467 master_runner.cc:186] Error getting master registration for 127.9.117.254:37039: OK, instance_id { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" instance_seqno: 1761250666322971 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20251023 20:17:46.790225 569 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:46.790268 569 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:46.790266 467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:17:46.790519 380 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } }
I20251023 20:17:46.790638 380 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6627fce0896445b1ba5927cdd86236e2, State: Initialized, Role: FOLLOWER
I20251023 20:17:46.790697 380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } }
I20251023 20:17:46.791025 571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.791088 380 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:46.791113 571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:46.789858 389 master_runner.cc:186] Error getting master registration for 127.9.117.254:37039: OK, instance_id { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" instance_seqno: 1761250666322971 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251023 20:17:46.792701 389 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:17:46.792902 354 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 20:17:46.793134 354 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:17:46.793264 582 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:46.793316 582 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:46.794163 354 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } }
I20251023 20:17:46.794435 354 leader_election.cc:290] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [CANDIDATE]: Term 1 election: Requested vote from peers 873e996872814db6aa71565842291c3d (127.9.117.253:43507), f608ef5afef646ee81696d82b7f9abe7 (127.9.117.252:35317)
I20251023 20:17:46.794867 431 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6627fce0896445b1ba5927cdd86236e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "873e996872814db6aa71565842291c3d"
I20251023 20:17:46.794981 431 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:46.795783 431 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6627fce0896445b1ba5927cdd86236e2 in term 1.
I20251023 20:17:46.796065 329 leader_election.cc:304] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6627fce0896445b1ba5927cdd86236e2, 873e996872814db6aa71565842291c3d; no voters:
I20251023 20:17:46.796233 571 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:46.796415 571 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [term 1 LEADER]: Becoming Leader. State: Replica: 6627fce0896445b1ba5927cdd86236e2, State: Running, Role: LEADER
I20251023 20:17:46.796523 571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } }
I20251023 20:17:46.797065 584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6627fce0896445b1ba5927cdd86236e2. Latest consensus state: current_term: 1 leader_uuid: "6627fce0896445b1ba5927cdd86236e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.797152 584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:46.797207 509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6627fce0896445b1ba5927cdd86236e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f608ef5afef646ee81696d82b7f9abe7"
I20251023 20:17:46.797360 509 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:46.799360 509 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6627fce0896445b1ba5927cdd86236e2 in term 1.
I20251023 20:17:46.801126 312 master_runner.cc:263] No leader master found from master 6627fce0896445b1ba5927cdd86236e2
I20251023 20:17:46.801329 312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:17:46.801880 590 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:46.802062 590 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:46.804226 509 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [term 1 FOLLOWER]: Refusing update from remote peer 6627fce0896445b1ba5927cdd86236e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:17:46.804639 571 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:17:46.806241 544 sys_catalog.cc:455] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6627fce0896445b1ba5927cdd86236e2. Latest consensus state: current_term: 1 leader_uuid: "6627fce0896445b1ba5927cdd86236e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.806354 544 sys_catalog.cc:458] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:46.806926 431 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [term 1 FOLLOWER]: Refusing update from remote peer 6627fce0896445b1ba5927cdd86236e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:17:46.807302 571 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:17:46.807451 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.797558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.9.117.254:37039)
1023 20:17:46.797607 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.797609 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.799405 (+ 1796us) server_negotiation.cc:408] Connection header received
1023 20:17:46.799516 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.799517 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.799527 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.799558 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.800617 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.801345 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.802133 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.802315 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.804072 (+ 1757us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.804077 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.804077 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.804086 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.806248 (+ 2162us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.806648 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.806649 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.806650 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.806675 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.806857 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.806858 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.806858 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.806982 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.807058 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.807267 (+ 209us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.807361 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251023 20:17:46.808898 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6627fce0896445b1ba5927cdd86236e2. Latest consensus state: current_term: 1 leader_uuid: "6627fce0896445b1ba5927cdd86236e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.809008 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:46.809598 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.799187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.9.117.253:43507)
1023 20:17:46.799237 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.799239 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.799844 (+ 605us) server_negotiation.cc:408] Connection header received
1023 20:17:46.800116 (+ 272us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.800117 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.800130 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.800163 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.801752 (+ 1589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.802523 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.804045 (+ 1522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.804210 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.805935 (+ 1725us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.805939 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.805940 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.805948 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.807863 (+ 1915us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.808707 (+ 844us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.808708 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.808709 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.808761 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.808953 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.808953 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.808954 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.809079 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.809165 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.809387 (+ 222us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.809487 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:17:46.810269 584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6627fce0896445b1ba5927cdd86236e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.810367 584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:46.810773 544 sys_catalog.cc:455] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6627fce0896445b1ba5927cdd86236e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.810861 544 sys_catalog.cc:458] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:46.811177 594 mvcc.cc:204] Tried to move back new op lower bound from 7214082731227504640 to 7214082731199557632. Current Snapshot: MvccSnapshot[applied={T|T < 7214082731227504640}]
I20251023 20:17:46.811722 590 catalog_manager.cc:1357] Generated new cluster ID: f8f8a27118934ecc92a41a9ed2ec0cd8
I20251023 20:17:46.811781 590 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:46.811872 595 mvcc.cc:204] Tried to move back new op lower bound from 7214082731227504640 to 7214082731199557632. Current Snapshot: MvccSnapshot[applied={T|T < 7214082731227504640}]
I20251023 20:17:46.813112 571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6627fce0896445b1ba5927cdd86236e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.813194 571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2 [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:46.813791 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6627fce0896445b1ba5927cdd86236e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6627fce0896445b1ba5927cdd86236e2" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 37039 } } peers { permanent_uuid: "873e996872814db6aa71565842291c3d" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 43507 } } peers { permanent_uuid: "f608ef5afef646ee81696d82b7f9abe7" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 35317 } } }
I20251023 20:17:46.813896 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:46.815091 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:46.799545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.9.117.252:35317)
1023 20:17:46.799589 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.799591 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.807798 (+ 8207us) server_negotiation.cc:408] Connection header received
1023 20:17:46.807905 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.807906 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.807921 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.807956 (+ 35us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.808698 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.809463 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:46.810238 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:46.810435 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:46.811950 (+ 1515us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:46.811954 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:46.811954 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:46.811963 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:46.813888 (+ 1925us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.814309 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.814310 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.814311 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.814337 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:46.814508 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:46.814509 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:46.814510 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:46.814644 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:46.814789 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:46.814899 (+ 110us) server_negotiation.cc:300] Negotiation successful
1023 20:17:46.814998 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:17:46.815335 597 mvcc.cc:204] Tried to move back new op lower bound from 7214082731227504640 to 7214082731199557632. Current Snapshot: MvccSnapshot[applied={T|T < 7214082731227504640 or (T in {7214082731227504640})}]
I20251023 20:17:46.822722 590 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:46.823772 590 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:46.839226 590 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6627fce0896445b1ba5927cdd86236e2: Generated new TSK 0
I20251023 20:17:46.839493 590 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251023 20:17:46.868530 383 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.867064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.9.117.254:37039)
1023 20:17:46.867117 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.867120 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.867814 (+ 694us) server_negotiation.cc:408] Connection header received
1023 20:17:46.867934 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.867936 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.867947 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.867979 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.868456 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42974: BlockingRecv error: recv got EOF from 127.0.0.1:42974 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20251023 20:17:46.869570 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.867504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.9.117.253:43507)
1023 20:17:46.867547 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.867550 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.869017 (+ 1467us) server_negotiation.cc:408] Connection header received
1023 20:17:46.869121 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.869122 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.869134 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.869164 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.869502 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52056: BlockingRecv error: recv got EOF from 127.0.0.1:52056 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251023 20:17:46.870826 541 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.870110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.9.117.252:35317)
1023 20:17:46.870154 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.870156 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.870268 (+ 112us) server_negotiation.cc:408] Connection header received
1023 20:17:46.870371 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.870373 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.870385 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.870416 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.870768 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50744: BlockingRecv error: recv got EOF from 127.0.0.1:50744 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20251023 20:17:46.873623 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.872700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.9.117.253:43507)
1023 20:17:46.872776 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.872778 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.873007 (+ 229us) server_negotiation.cc:408] Connection header received
1023 20:17:46.873139 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.873140 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.873160 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.873191 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.873564 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52064: BlockingRecv error: recv got EOF from 127.0.0.1:52064 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20251023 20:17:46.877003 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.876148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.9.117.253:43507)
1023 20:17:46.876185 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.876187 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.876430 (+ 243us) server_negotiation.cc:408] Connection header received
1023 20:17:46.876535 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.876536 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.876546 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.876576 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.876948 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52078: BlockingRecv error: recv got EOF from 127.0.0.1:52078 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20251023 20:17:46.882014 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.880908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.9.117.253:43507)
1023 20:17:46.880952 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.880954 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.881220 (+ 266us) server_negotiation.cc:408] Connection header received
1023 20:17:46.881326 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.881327 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.881338 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.881369 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.881953 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52094: BlockingRecv error: recv got EOF from 127.0.0.1:52094 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20251023 20:17:46.888564 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.887634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.9.117.253:43507)
1023 20:17:46.887680 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.887682 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.887955 (+ 273us) server_negotiation.cc:408] Connection header received
1023 20:17:46.888064 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.888065 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.888079 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.888110 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.888488 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52100: BlockingRecv error: recv got EOF from 127.0.0.1:52100 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251023 20:17:46.899353 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.898274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.9.117.253:43507)
1023 20:17:46.898323 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.898326 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.898632 (+ 306us) server_negotiation.cc:408] Connection header received
1023 20:17:46.898746 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.898748 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.898775 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.898807 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.899283 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52116: BlockingRecv error: recv got EOF from 127.0.0.1:52116 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20251023 20:17:46.918357 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.917302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.9.117.253:43507)
1023 20:17:46.917368 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.917370 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.917708 (+ 338us) server_negotiation.cc:408] Connection header received
1023 20:17:46.917823 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.917824 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.917837 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.917870 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.918288 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52122: BlockingRecv error: recv got EOF from 127.0.0.1:52122 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20251023 20:17:46.953547 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:46.952365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.9.117.253:43507)
1023 20:17:46.952416 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:46.952419 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:46.952774 (+ 355us) server_negotiation.cc:408] Connection header received
1023 20:17:46.952910 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:46.952912 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:46.952925 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:46.952957 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:46.953401 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52136: BlockingRecv error: recv got EOF from 127.0.0.1:52136 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20251023 20:17:47.020449 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:47.019408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.9.117.253:43507)
1023 20:17:47.019457 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:47.019459 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:47.019787 (+ 328us) server_negotiation.cc:408] Connection header received
1023 20:17:47.019893 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:47.019894 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:47.019907 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:47.019937 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:47.020360 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52138: BlockingRecv error: recv got EOF from 127.0.0.1:52138 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251023 20:17:47.151072 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:47.150351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.9.117.253:43507)
1023 20:17:47.150469 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:47.150471 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:47.150507 (+ 36us) server_negotiation.cc:408] Connection header received
1023 20:17:47.150536 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:47.150537 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:47.150548 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:47.150596 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:47.151015 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52148: BlockingRecv error: recv got EOF from 127.0.0.1:52148 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20251023 20:17:47.409906 463 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:47.408817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.9.117.253:43507)
1023 20:17:47.408962 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:47.408965 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:47.408975 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:17:47.409006 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:47.409007 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:47.409018 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:47.409082 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:47.409666 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52152: BlockingRecv error: recv got EOF from 127.0.0.1:52152 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20251023 20:17:47.789539 555 catalog_manager.cc:1269] Loaded cluster ID: f8f8a27118934ecc92a41a9ed2ec0cd8
I20251023 20:17:47.789979 555 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7: loading cluster ID for follower catalog manager: success
I20251023 20:17:47.790763 569 catalog_manager.cc:1269] Loaded cluster ID: f8f8a27118934ecc92a41a9ed2ec0cd8
I20251023 20:17:47.790827 569 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d: loading cluster ID for follower catalog manager: success
I20251023 20:17:47.791121 555 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7: acquiring CA information for follower catalog manager: success
I20251023 20:17:47.791436 555 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f608ef5afef646ee81696d82b7f9abe7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:17:47.792246 569 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d: acquiring CA information for follower catalog manager: success
I20251023 20:17:47.792582 569 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 873e996872814db6aa71565842291c3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:17:47.926196 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:47.923133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52168 (local address 127.9.117.253:43507)
1023 20:17:47.923254 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:47.923256 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:47.923266 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:17:47.923296 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:47.923296 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:47.923347 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:47.923420 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:47.923835 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.924534 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:47.925527 (+ 993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.925715 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:47.925780 (+ 65us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:47.925906 (+ 126us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:47.925986 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:47.926047 (+ 61us) server_negotiation.cc:300] Negotiation successful
1023 20:17:47.926089 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20251023 20:17:47.931102 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:47.928703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.9.117.252:35317)
1023 20:17:47.928892 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:47.928895 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:47.928904 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:17:47.928955 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:47.928956 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:47.928986 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:47.929059 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:47.929341 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.929926 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:47.930540 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.930685 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:47.930729 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:47.930844 (+ 115us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:47.930922 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:47.930961 (+ 39us) server_negotiation.cc:300] Negotiation successful
1023 20:17:47.931000 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20251023 20:17:47.936146 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:47.933086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.9.117.252:35317)
1023 20:17:47.933128 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:47.933130 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:47.933378 (+ 248us) server_negotiation.cc:408] Connection header received
1023 20:17:47.933485 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:47.933486 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:47.933502 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:47.933565 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:47.933994 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.934859 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:47.935574 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.935746 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:47.935769 (+ 23us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:47.935798 (+ 29us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:47.935880 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:47.936008 (+ 128us) server_negotiation.cc:300] Negotiation successful
1023 20:17:47.936039 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251023 20:17:47.936305 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:47.933205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.9.117.254:37039)
1023 20:17:47.933533 (+ 328us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:47.933536 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:47.933552 (+ 16us) server_negotiation.cc:408] Connection header received
1023 20:17:47.933613 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:47.933613 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:47.933654 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:47.933738 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:47.934104 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.934670 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:47.935439 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.935665 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:47.935738 (+ 73us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:47.935872 (+ 134us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:47.935946 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:47.936185 (+ 239us) server_negotiation.cc:300] Negotiation successful
1023 20:17:47.936221 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":70,"threads_started":1}
I20251023 20:17:47.936782 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:47.933030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.9.117.253:43507)
1023 20:17:47.933103 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:47.933105 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:47.933385 (+ 280us) server_negotiation.cc:408] Connection header received
1023 20:17:47.933444 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:47.933445 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:47.933465 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:47.933530 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:47.933910 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.934867 (+ 957us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:47.936155 (+ 1288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:47.936343 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:47.936444 (+ 101us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:47.936477 (+ 33us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:47.936519 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:47.936634 (+ 115us) server_negotiation.cc:300] Negotiation successful
1023 20:17:47.936674 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251023 20:17:47.938601 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 312
I20251023 20:17:47.945201 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 389
I20251023 20:17:47.951213 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 467
2025-10-23T20:17:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3975 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 23 20:17:48 dist-test-slave-x62p krb5kdc[682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 23 20:17:48 dist-test-slave-x62p krb5kdc[682](info): set up 2 sockets
Oct 23 20:17:48 dist-test-slave-x62p krb5kdc[682](info): commencing operation
krb5kdc: starting...
W20251023 20:17:49.998019 9687 mini_kdc.cc:121] Time spent starting KDC: real 2.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/test-user.keytab.
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-23T20:17:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:50.157971 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:45295
--webserver_interface=127.9.117.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:45295,127.9.117.253:39229,127.9.117.252:37121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251023 20:17:50.243700 698 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251023 20:17:50.243904 698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:50.243921 698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:50.243942 698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:50.245502 698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:50.245548 698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:50.245563 698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:50.245576 698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:50.245591 698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:50.247220 698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:45295,127.9.117.253:39229,127.9.117.252:37121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.254:45295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.117.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:50.247432 698 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:50.247634 698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:50.250278 706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:50.250401 698 server_base.cc:1047] running on GCE node
W20251023 20:17:50.250259 703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:50.250278 704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:50.250720 698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:50.250931 698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:50.252079 698 hybrid_clock.cc:648] HybridClock initialized: now 1761250670252069 us; error 28 us; skew 500 ppm
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:50.254580 698 init.cc:377] Logged in from keytab as kudu/127.9.117.254@KRBTEST.COM (short username kudu)
I20251023 20:17:50.255141 698 webserver.cc:492] Webserver started at http://127.9.117.254:45505/ using document root <none> and password file <none>
I20251023 20:17:50.255333 698 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:50.255376 698 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:50.255501 698 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:50.256489 698 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/data/instance:
uuid: "e601c89a251b4d228dbb0d4dab079b7d"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "6c1f0561938b531a6a1d59227c6105ae"
server_key_iv: "65ab67f37603a88d370782bbfc97983f"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.256860 698 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/wal/instance:
uuid: "e601c89a251b4d228dbb0d4dab079b7d"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "6c1f0561938b531a6a1d59227c6105ae"
server_key_iv: "65ab67f37603a88d370782bbfc97983f"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.258103 698 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:50.258769 713 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:50.258946 698 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251023 20:17:50.259023 698 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/wal
uuid: "e601c89a251b4d228dbb0d4dab079b7d"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "6c1f0561938b531a6a1d59227c6105ae"
server_key_iv: "65ab67f37603a88d370782bbfc97983f"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.259086 698 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:50.272221 698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:50.278285 698 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:50.278448 698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:50.283607 698 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.254:45295
I20251023 20:17:50.283749 765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.254:45295 every 8 connection(s)
I20251023 20:17:50.284024 698 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/data/info.pb
I20251023 20:17:50.284703 766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:50.286748 766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:50.293284 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 698
I20251023 20:17:50.293395 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-0/wal/instance
I20251023 20:17:50.293577 9687 external_mini_cluster.cc:1468] Setting key 46352f4bb9a1793040377308564b2f84
I20251023 20:17:50.297161 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.287474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:45295 (local address 127.0.0.1:42824)
1023 20:17:50.287822 (+ 348us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.287856 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.288356 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.288859 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.288862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.289204 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.289464 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.289467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.290547 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.290548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.290922 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.290924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.291041 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.291724 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.291744 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.293298 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.295228 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.295236 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.295239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.295667 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.295940 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.295941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.295941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.296025 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.296439 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.296445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.296532 (+ 87us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.296645 (+ 113us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.296892 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":244,"threads_started":1}
I20251023 20:17:50.297559 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.287548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.9.117.254:45295)
1023 20:17:50.287871 (+ 323us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.287879 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.287957 (+ 78us) server_negotiation.cc:408] Connection header received
1023 20:17:50.288558 (+ 601us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.288561 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.288595 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.288834 (+ 239us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.289601 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.290409 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.291017 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.291359 (+ 342us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.293400 (+ 2041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.293408 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.293408 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.293415 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.295135 (+ 1720us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.295808 (+ 673us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.295812 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.295812 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.295866 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.296142 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.296143 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.296144 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.296368 (+ 224us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.296455 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.296707 (+ 252us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.297103 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":239,"server-negotiator.queue_time_us":294,"thread_start_us":227,"threads_started":1}
I20251023 20:17:50.298287 766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:50.298722 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.298554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35442)
1023 20:17:50.298617 (+ 63us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.298682 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251023 20:17:50.298903 717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
I20251023 20:17:50.299479 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.299350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35448)
1023 20:17:50.299417 (+ 67us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.299451 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251023 20:17:50.299667 766 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:39229: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.9.117.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:50.348884 766 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } attempt: 1
I20251023 20:17:50.349086 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.253:39229
--webserver_interface=127.9.117.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:45295,127.9.117.253:39229,127.9.117.252:37121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:17:50.349390 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.349237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35452)
1023 20:17:50.349322 (+ 85us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.349353 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20251023 20:17:50.350075 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.349842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35458)
1023 20:17:50.349890 (+ 48us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.350043 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251023 20:17:50.350207 766 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:39229: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
I20251023 20:17:50.418390 766 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } attempt: 2
I20251023 20:17:50.419005 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.418797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35470)
1023 20:17:50.418888 (+ 91us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.418950 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20251023 20:17:50.419415 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.419311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35480)
1023 20:17:50.419351 (+ 40us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.419388 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20251023 20:17:50.419531 766 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.253:39229: Network error: Client connection negotiation failed: client connection to 127.9.117.253:39229: connect: Connection refused (error 111)
W20251023 20:17:50.439549 775 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251023 20:17:50.439771 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:50.439790 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:50.439815 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:50.441663 775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:50.441730 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:50.441744 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:50.441758 775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:50.441773 775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:50.444442 775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:45295,127.9.117.253:39229,127.9.117.252:37121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.253:39229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.117.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:50.445120 775 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:50.445379 775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:50.448350 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:50.448516 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:50.448549 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:50.448850 775 server_base.cc:1047] running on GCE node
I20251023 20:17:50.449103 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:50.449354 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:50.450536 775 hybrid_clock.cc:648] HybridClock initialized: now 1761250670450522 us; error 29 us; skew 500 ppm
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:50.453382 775 init.cc:377] Logged in from keytab as kudu/127.9.117.253@KRBTEST.COM (short username kudu)
I20251023 20:17:50.454310 775 webserver.cc:492] Webserver started at http://127.9.117.253:45053/ using document root <none> and password file <none>
I20251023 20:17:50.454530 775 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:50.454586 775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:50.454715 775 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:50.455935 775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/data/instance:
uuid: "4c7aa04201f14130b97531bd4d75719f"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "c68b9d1ac39cf820d82de15d6aff70de"
server_key_iv: "201a18eef7e53ed44e45eed350b7adb8"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.456284 775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/wal/instance:
uuid: "4c7aa04201f14130b97531bd4d75719f"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "c68b9d1ac39cf820d82de15d6aff70de"
server_key_iv: "201a18eef7e53ed44e45eed350b7adb8"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.457756 775 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251023 20:17:50.458550 790 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:50.458843 775 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251023 20:17:50.458930 775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/wal
uuid: "4c7aa04201f14130b97531bd4d75719f"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "c68b9d1ac39cf820d82de15d6aff70de"
server_key_iv: "201a18eef7e53ed44e45eed350b7adb8"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.459007 775 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:50.474851 775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:50.477821 775 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:50.477977 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:50.484268 775 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.253:39229
I20251023 20:17:50.484293 842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.253:39229 every 8 connection(s)
I20251023 20:17:50.484795 775 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/data/info.pb
I20251023 20:17:50.485476 843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:50.487459 843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:50.494913 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 775
I20251023 20:17:50.495035 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-1/wal/instance
I20251023 20:17:50.495204 9687 external_mini_cluster.cc:1468] Setting key eca1b730e9b6d20af207cb7740d55af4
I20251023 20:17:50.497920 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.488314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.9.117.254:45295)
1023 20:17:50.488375 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.488377 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.488589 (+ 212us) server_negotiation.cc:408] Connection header received
1023 20:17:50.489043 (+ 454us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.489045 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.489060 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.489128 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.490292 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.490978 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.491733 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.491922 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.494108 (+ 2186us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.494115 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.494116 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.494125 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.496328 (+ 2203us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.496962 (+ 634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.496965 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.496965 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.496998 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.497206 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.497207 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.497207 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.497329 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.497393 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.497694 (+ 301us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.497815 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251023 20:17:50.498054 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.488231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:45295 (local address 127.0.0.1:42840)
1023 20:17:50.488432 (+ 201us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.488464 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.488916 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.489240 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.489245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.489747 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.490069 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.490073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.491203 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.491205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.491599 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.491601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.491712 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.492327 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.492350 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.493988 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.496536 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.496548 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.496552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.496879 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.497059 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.497060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.497060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.497135 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.497386 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.497390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.497546 (+ 156us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.497652 (+ 106us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.497828 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20251023 20:17:50.498699 843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.117.252@KRBTEST.COM; defaulting to no policy
I20251023 20:17:50.506754 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.499329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35490)
1023 20:17:50.499489 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.499502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.499551 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.499838 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.499839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.499993 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.500140 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.500142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.501054 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.501055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.501396 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.501398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.501474 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.502102 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.502107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.503648 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.505812 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.505814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.505814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.505956 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.506132 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.506133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.506133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.506160 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.506398 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.506399 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.506436 (+ 37us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.506513 (+ 77us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.506635 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20251023 20:17:50.506990 847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.499442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.9.117.253:39229)
1023 20:17:50.499704 (+ 262us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.499716 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.499732 (+ 16us) server_negotiation.cc:408] Connection header received
1023 20:17:50.499750 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.499752 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.499763 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.499826 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.500216 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.500969 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.501468 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.501704 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.503767 (+ 2063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.503779 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.503780 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.503787 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.505707 (+ 1920us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.506039 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.506041 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.506042 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.506075 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.506214 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.506215 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.506215 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.506341 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.506474 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.506544 (+ 70us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.506734 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":226,"thread_start_us":154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.117.252@KRBTEST.COM" created.
I20251023 20:17:50.507790 843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:50.508216 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.508058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43178)
1023 20:17:50.508123 (+ 65us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.508182 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251023 20:17:50.508387 791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
I20251023 20:17:50.508980 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.508884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43182)
1023 20:17:50.508932 (+ 48us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.508960 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251023 20:17:50.509131 843 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:37121: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
I20251023 20:17:50.510705 766 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } attempt: 3
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
I20251023 20:17:50.518195 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.511053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35498)
1023 20:17:50.511109 (+ 56us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.511121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.511173 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.511317 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.511318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.511503 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.511694 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.511696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.512613 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.512614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.513045 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.513047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.513121 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.513604 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.513608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.514886 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.517215 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.517217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.517217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.517389 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.517567 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.517568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.517568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.517598 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.517828 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.517830 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.517899 (+ 69us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.517960 (+ 61us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.518088 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20251023 20:17:50.518236 847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.511111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.9.117.253:39229)
1023 20:17:50.511169 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.511172 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.511183 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:17:50.511237 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.511238 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.511251 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.511305 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.511777 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.512475 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.513132 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.513373 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.514983 (+ 1610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.514989 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.514989 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.514999 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.517117 (+ 2118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.517468 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.517470 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.517470 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.517498 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.517650 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.517651 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.517652 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.517766 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.517832 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.517993 (+ 161us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.518111 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251023 20:17:50.519074 766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:50.519547 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.519345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43186)
1023 20:17:50.519474 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.519514 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20251023 20:17:50.520398 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.520045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43188)
1023 20:17:50.520340 (+ 295us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.520372 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279}
W20251023 20:17:50.520627 766 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:37121: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.117.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.9.117.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.9.117.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.117.252@KRBTEST.COM" created.
I20251023 20:17:50.551810 766 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } attempt: 1
I20251023 20:17:50.552317 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.552178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43196)
1023 20:17:50.552249 (+ 71us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.552281 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20251023 20:17:50.552670 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.552569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43202)
1023 20:17:50.552612 (+ 43us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.552638 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20251023 20:17:50.552822 766 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:37121: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
I20251023 20:17:50.558413 843 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } attempt: 1
I20251023 20:17:50.559002 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.558798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43208)
1023 20:17:50.558931 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.558965 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20251023 20:17:50.559763 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.559561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43218)
1023 20:17:50.559691 (+ 130us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.559735 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20251023 20:17:50.559913 843 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:37121: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.117.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.9.117.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab.
I20251023 20:17:50.561820 9687 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskut_hJr/build/release/bin/kudu
/tmp/dist-test-taskut_hJr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.252:37121
--webserver_interface=127.9.117.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.9.117.212:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.117.254:45295,127.9.117.253:39229,127.9.117.252:37121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251023 20:17:50.628021 766 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } attempt: 2
I20251023 20:17:50.628122 843 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } attempt: 2
I20251023 20:17:50.628563 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.628393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43220)
1023 20:17:50.628476 (+ 83us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.628517 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20251023 20:17:50.628564 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.628421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43232)
1023 20:17:50.628473 (+ 52us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.628518 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20251023 20:17:50.628973 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.628867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43248)
1023 20:17:50.628921 (+ 54us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.628947 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20251023 20:17:50.628973 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.628875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43264)
1023 20:17:50.628917 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.628945 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20251023 20:17:50.629073 766 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:37121: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
W20251023 20:17:50.629146 843 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.117.252:37121: Network error: Client connection negotiation failed: client connection to 127.9.117.252:37121: connect: Connection refused (error 111)
W20251023 20:17:50.653496 852 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251023 20:17:50.653716 852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:17:50.653741 852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:17:50.653769 852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:17:50.655342 852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:17:50.655395 852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251023 20:17:50.655409 852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:17:50.655423 852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:17:50.655436 852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:17:50.657584 852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.117.212:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.117.254:45295,127.9.117.253:39229,127.9.117.252:37121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/krb5kdc/kudu.keytab
--principal=kudu/127.9.117.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.117.252:37121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.117.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type RELEASE
built by None at 23 Oct 2025 19:57:35 UTC on 5fd53c4cbb9d
build id 8683
I20251023 20:17:50.657841 852 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:17:50.658005 852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:17:50.661252 860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:50.661271 857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:17:50.661252 858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:17:50.661291 852 server_base.cc:1047] running on GCE node
I20251023 20:17:50.661845 852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:17:50.662111 852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:17:50.663288 852 hybrid_clock.cc:648] HybridClock initialized: now 1761250670663269 us; error 36 us; skew 500 ppm
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251023 20:17:50.666733 852 init.cc:377] Logged in from keytab as kudu/127.9.117.252@KRBTEST.COM (short username kudu)
I20251023 20:17:50.667394 852 webserver.cc:492] Webserver started at http://127.9.117.252:32965/ using document root <none> and password file <none>
I20251023 20:17:50.667596 852 fs_manager.cc:362] Metadata directory not provided
I20251023 20:17:50.667636 852 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:17:50.667760 852 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:17:50.668988 852 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/data/instance:
uuid: "fbc722591ec44cc4ad46f312cacfc686"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "31e0a10207f15e610bd702bfb3a3e787"
server_key_iv: "3ed6f1015ce9949a208d7d071329d859"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.669314 852 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/wal/instance:
uuid: "fbc722591ec44cc4ad46f312cacfc686"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "31e0a10207f15e610bd702bfb3a3e787"
server_key_iv: "3ed6f1015ce9949a208d7d071329d859"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.670852 852 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251023 20:17:50.671749 867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:17:50.672032 852 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251023 20:17:50.672122 852 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/data,/tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/wal
uuid: "fbc722591ec44cc4ad46f312cacfc686"
format_stamp: "Formatted at 2025-10-23 20:17:50 on dist-test-slave-x62p"
server_key: "31e0a10207f15e610bd702bfb3a3e787"
server_key_iv: "3ed6f1015ce9949a208d7d071329d859"
server_key_version: "encryptionkey@0"
I20251023 20:17:50.672199 852 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:17:50.687268 852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:17:50.689841 852 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:17:50.690009 852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:17:50.694482 852 rpc_server.cc:307] RPC server started. Bound to: 127.9.117.252:37121
I20251023 20:17:50.694564 919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.117.252:37121 every 8 connection(s)
I20251023 20:17:50.694900 852 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/data/info.pb
I20251023 20:17:50.695535 920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:17:50.697937 9687 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskut_hJr/build/release/bin/kudu as pid 852
I20251023 20:17:50.698074 9687 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskut_hJr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761250300132822-9687-0/minicluster-data/master-2/wal/instance
I20251023 20:17:50.698242 9687 external_mini_cluster.cc:1468] Setting key 1bca8b282ddb744b21fd28959989cdad
I20251023 20:17:50.698047 920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:17:50.708587 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.699009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.9.117.254:45295)
1023 20:17:50.699065 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.699068 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.699269 (+ 201us) server_negotiation.cc:408] Connection header received
1023 20:17:50.699790 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.699791 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.699808 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.699888 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.700628 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.701127 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.701890 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.702075 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.705209 (+ 3134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.705216 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.705217 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.705228 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.707017 (+ 1789us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.707493 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.707494 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.707495 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.707524 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.707772 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.707773 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.707773 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.707919 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.708002 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.708387 (+ 385us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.708482 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251023 20:17:50.708776 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.698931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.254:45295 (local address 127.0.0.1:42856)
1023 20:17:50.699122 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.699153 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.699582 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.699946 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.699948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.700269 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.700524 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.700527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.701204 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.701205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.701801 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.701804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.701904 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.702897 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.702926 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.704952 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.707120 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.707127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.707130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.707366 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.707597 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.707598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.707599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.707683 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.708029 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.708034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.708238 (+ 204us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.708344 (+ 106us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.708507 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":125,"threads_started":1}
I20251023 20:17:50.709525 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.698911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.9.117.252:37121)
1023 20:17:50.699136 (+ 225us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.699155 (+ 19us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.699195 (+ 40us) server_negotiation.cc:408] Connection header received
1023 20:17:50.699598 (+ 403us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.699608 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.699648 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.699736 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.700602 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.701544 (+ 942us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.702248 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.702595 (+ 347us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.705698 (+ 3103us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.705710 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.705710 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.705719 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.707676 (+ 1957us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.708108 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.708111 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.708111 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.708149 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.708351 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.708352 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.708353 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.708472 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.708610 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.709098 (+ 488us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.709354 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20251023 20:17:50.709570 920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } has no permanent_uuid. Determining permanent_uuid...
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.254@KRBTEST.COM
I20251023 20:17:50.718214 847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.709938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.9.117.253:39229)
1023 20:17:50.710005 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.710008 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.710040 (+ 32us) server_negotiation.cc:408] Connection header received
1023 20:17:50.710111 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.710112 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.710125 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.710187 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.710701 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.711373 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.712117 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.712293 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.714369 (+ 2076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.714375 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.714375 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.714383 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.716610 (+ 2227us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.717099 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.717101 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.717101 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.717132 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.717558 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.717559 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.717560 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.717723 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.717781 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.717996 (+ 215us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.718109 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251023 20:17:50.718310 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.710365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.9.117.254:45295)
1023 20:17:50.710414 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.710416 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.710425 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:17:50.710475 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.710476 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.710486 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.710546 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.711250 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.711875 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.712508 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.712683 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.715030 (+ 2347us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.715034 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.715035 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.715043 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.717058 (+ 2015us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.717562 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.717564 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.717564 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.717592 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.717828 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.717828 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.717829 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.717919 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.718080 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.718119 (+ 39us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.718209 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251023 20:17:50.718189 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.709867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.253:39229 (local address 127.0.0.1:35504)
1023 20:17:50.709994 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.710005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.710054 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.710225 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.710226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.710398 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.710620 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.710622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.711473 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.711474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.712034 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.712036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.712098 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.712626 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.712632 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.714224 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.716828 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.716829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.716830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.717009 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.717328 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.717329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.717329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.717373 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.717823 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.717825 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.717885 (+ 60us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.717947 (+ 62us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.718077 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20251023 20:17:50.719255 920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:17:50.720322 843 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } attempt: 3
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.252@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.117.253@KRBTEST.COM
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.253@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:17:50.727788 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.719641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.9.117.252:37121)
1023 20:17:50.719675 (+ 34us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.719676 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.719718 (+ 42us) server_negotiation.cc:408] Connection header received
1023 20:17:50.719801 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.719802 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.719809 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.719845 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.720340 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.721190 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.721961 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.722143 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.724590 (+ 2447us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.724594 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.724594 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.724603 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.726696 (+ 2093us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.727019 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.727021 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.727021 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.727055 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.727240 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.727241 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.727242 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.727342 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.727406 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.727553 (+ 147us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.727651 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251023 20:17:50.728132 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.719563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43282)
1023 20:17:50.719659 (+ 96us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.719670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.719735 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.719864 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.719865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.720040 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.720234 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.720236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.721347 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.721349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.721867 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.721869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.722543 (+ 674us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.723033 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.723038 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.724502 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.726780 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.726781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.726782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.726933 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.727142 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.727142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.727143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.727177 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.727398 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.727399 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.727434 (+ 35us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.727938 (+ 504us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.728053 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20251023 20:17:50.728206 766 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } attempt: 3
I20251023 20:17:50.729703 847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.719201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.9.117.253:39229)
1023 20:17:50.719445 (+ 244us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.719447 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.719456 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:17:50.719478 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.719478 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.719490 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.719549 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.720368 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.721001 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.721752 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.721951 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.725916 (+ 3965us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.725920 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.725921 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.725928 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.727837 (+ 1909us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.728780 (+ 943us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.728782 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.728782 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.728811 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.728989 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.728989 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.728990 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.729121 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.729253 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.729477 (+ 224us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.729594 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20251023 20:17:50.729775 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.720583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43296)
1023 20:17:50.720702 (+ 119us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.720723 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.720819 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.721409 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.721410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.721560 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.721678 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.721680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.722298 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.722299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.722643 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.722644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.722723 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.723261 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.723265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.725645 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.728834 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.728835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.728835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.729032 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.729207 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.729208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.729208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.729233 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.729500 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.729501 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.729535 (+ 34us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.729583 (+ 48us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.729667 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20251023 20:17:50.730083 920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686: Bootstrap starting.
I20251023 20:17:50.730398 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.720638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.9.117.252:37121)
1023 20:17:50.721296 (+ 658us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.721297 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.721306 (+ 9us) server_negotiation.cc:408] Connection header received
1023 20:17:50.721320 (+ 14us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.721320 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.721328 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.721382 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.721766 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.722229 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.722878 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.723090 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.725733 (+ 2643us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.725740 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.725740 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.725747 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.728508 (+ 2761us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.729111 (+ 603us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.729113 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.729113 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.729137 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.729300 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.729301 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.729301 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.729428 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.729845 (+ 417us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.729887 (+ 42us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.729972 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":148,"threads_started":1}
I20251023 20:17:50.730829 920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:50.731173 920 log.cc:826] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686: Log is configured to *not* fsync() on all Append() calls
I20251023 20:17:50.732113 920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686: No bootstrap required, opened a new log
I20251023 20:17:50.732203 843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f: Bootstrap starting.
I20251023 20:17:50.733371 843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:50.733886 843 log.cc:826] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f: Log is configured to *not* fsync() on all Append() calls
Oct 23 20:17:50 dist-test-slave-x62p krb5kdc[682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761250670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.117.254@KRBTEST.COM for kudu/127.9.117.252@KRBTEST.COM
I20251023 20:17:50.734555 920 raft_consensus.cc:359] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } }
I20251023 20:17:50.735107 920 raft_consensus.cc:740] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc722591ec44cc4ad46f312cacfc686, State: Initialized, Role: FOLLOWER
I20251023 20:17:50.735193 843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f: No bootstrap required, opened a new log
I20251023 20:17:50.735262 920 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } }
I20251023 20:17:50.735409 9687 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251023 20:17:50.735742 920 sys_catalog.cc:565] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:50.738087 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.728519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.117.252:37121 (local address 127.0.0.1:43310)
1023 20:17:50.728613 (+ 94us) negotiation.cc:107] Waiting for socket to connect
1023 20:17:50.728625 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1023 20:17:50.728672 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:17:50.728919 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1023 20:17:50.728920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1023 20:17:50.729102 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1023 20:17:50.729286 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.729288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.730263 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.730264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1023 20:17:50.730808 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1023 20:17:50.730812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.730899 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.732931 (+ 2032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1023 20:17:50.732936 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1023 20:17:50.734464 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1023 20:17:50.736897 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.736898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.736899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.737139 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.737354 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1023 20:17:50.737355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1023 20:17:50.737355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1023 20:17:50.737396 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1023 20:17:50.737736 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1023 20:17:50.737737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1023 20:17:50.737802 (+ 65us) client_negotiation.cc:769] Sending connection context
1023 20:17:50.737870 (+ 68us) client_negotiation.cc:240] Negotiation successful
1023 20:17:50.737996 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20251023 20:17:50.738008 843 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } }
I20251023 20:17:50.738166 843 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7aa04201f14130b97531bd4d75719f, State: Initialized, Role: FOLLOWER
I20251023 20:17:50.736451 740 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e601c89a251b4d228dbb0d4dab079b7d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42868
I20251023 20:17:50.738230 843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } }
W20251023 20:17:50.738364 939 catalog_manager.cc:1568] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:50.738469 939 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:50.738571 843 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:50.738857 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.728552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.9.117.252:37121)
1023 20:17:50.728688 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.728690 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.728701 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:17:50.728721 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.728722 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.728756 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.728872 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.729379 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.730098 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.730942 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.731138 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.734573 (+ 3435us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.734577 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.734577 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.734585 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.736788 (+ 2203us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.737234 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.737236 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.737237 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.737265 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.737467 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.737468 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.737468 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.737633 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.737707 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.738657 (+ 950us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.738769 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20251023 20:17:50.738566 941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.739959 941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:50.740468 766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d: Bootstrap starting.
I20251023 20:17:50.741261 766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d: Neither blocks nor log segments found. Creating new log.
I20251023 20:17:50.741604 766 log.cc:826] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d: Log is configured to *not* fsync() on all Append() calls
W20251023 20:17:50.741622 952 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:50.741681 952 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:50.742336 852 master_runner.cc:186] Error getting master registration for 127.9.117.254:45295: OK, instance_id { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" instance_seqno: 1761250670281671 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251023 20:17:50.744590 852 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:17:50.742666 775 master_runner.cc:186] Error getting master registration for 127.9.117.254:45295: OK, instance_id { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" instance_seqno: 1761250670281671 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251023 20:17:50.744814 775 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:17:50.744951 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.745110 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:50.745500 766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d: No bootstrap required, opened a new log
I20251023 20:17:50.746759 766 raft_consensus.cc:359] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } }
I20251023 20:17:50.746999 766 raft_consensus.cc:740] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e601c89a251b4d228dbb0d4dab079b7d, State: Initialized, Role: FOLLOWER
I20251023 20:17:50.747079 766 consensus_queue.cc:260] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } }
I20251023 20:17:50.747434 766 sys_catalog.cc:565] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:17:50.747514 740 raft_consensus.cc:493] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20251023 20:17:50.747601 740 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:50.748701 740 raft_consensus.cc:515] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } }
I20251023 20:17:50.748994 740 leader_election.cc:290] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [CANDIDATE]: Term 1 election: Requested vote from peers 4c7aa04201f14130b97531bd4d75719f (127.9.117.253:39229), fbc722591ec44cc4ad46f312cacfc686 (127.9.117.252:37121)
I20251023 20:17:50.749629 817 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e601c89a251b4d228dbb0d4dab079b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4c7aa04201f14130b97531bd4d75719f"
I20251023 20:17:50.749759 817 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:17:50.750670 894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e601c89a251b4d228dbb0d4dab079b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fbc722591ec44cc4ad46f312cacfc686"
I20251023 20:17:50.750795 894 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:17:50.750851 966 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:17:50.750902 966 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:17:50.750946 817 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e601c89a251b4d228dbb0d4dab079b7d in term 1.
I20251023 20:17:50.751677 717 leader_election.cc:304] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c7aa04201f14130b97531bd4d75719f, e601c89a251b4d228dbb0d4dab079b7d; no voters:
I20251023 20:17:50.751842 894 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e601c89a251b4d228dbb0d4dab079b7d in term 1.
I20251023 20:17:50.747430 955 sys_catalog.cc:455] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.752445 955 sys_catalog.cc:458] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:50.752544 955 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:17:50.752666 955 raft_consensus.cc:697] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [term 1 LEADER]: Becoming Leader. State: Replica: e601c89a251b4d228dbb0d4dab079b7d, State: Running, Role: LEADER
I20251023 20:17:50.752836 955 consensus_queue.cc:237] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } }
I20251023 20:17:50.753377 698 master_runner.cc:263] No leader master found from master e601c89a251b4d228dbb0d4dab079b7d
I20251023 20:17:50.753433 698 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:17:50.753355 955 sys_catalog.cc:455] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e601c89a251b4d228dbb0d4dab079b7d. Latest consensus state: current_term: 1 leader_uuid: "e601c89a251b4d228dbb0d4dab079b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.753588 955 sys_catalog.cc:458] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:50.756202 974 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:17:50.756606 974 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:17:50.762449 894 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [term 1 FOLLOWER]: Refusing update from remote peer e601c89a251b4d228dbb0d4dab079b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:17:50.762781 817 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [term 1 FOLLOWER]: Refusing update from remote peer e601c89a251b4d228dbb0d4dab079b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:17:50.763165 955 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:17:50.763717 955 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:17:50.765620 941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e601c89a251b4d228dbb0d4dab079b7d. Latest consensus state: current_term: 1 leader_uuid: "e601c89a251b4d228dbb0d4dab079b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.765746 941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:50.766592 847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.754969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.9.117.253:39229)
1023 20:17:50.755013 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.755015 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.757633 (+ 2618us) server_negotiation.cc:408] Connection header received
1023 20:17:50.757756 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.757758 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.757771 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.757803 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.759116 (+ 1313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.759847 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.760670 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.760872 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.763027 (+ 2155us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.763031 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.763031 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.763058 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.764962 (+ 1904us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.765445 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.765446 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.765447 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.765476 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.765670 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.765671 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.765671 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.765814 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.765895 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.766390 (+ 495us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.766498 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:17:50.767095 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e601c89a251b4d228dbb0d4dab079b7d. Latest consensus state: current_term: 1 leader_uuid: "e601c89a251b4d228dbb0d4dab079b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.767201 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:50.773013 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.754111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.9.117.254:45295)
1023 20:17:50.754158 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.754160 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.757181 (+ 3021us) server_negotiation.cc:408] Connection header received
1023 20:17:50.765365 (+ 8184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.765367 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.765378 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.765410 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.766246 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.766962 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.767744 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.767945 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.769541 (+ 1596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.769547 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.769547 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.769556 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.771524 (+ 1968us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.771917 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.771918 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.771919 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.771946 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.772109 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.772110 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.772110 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.772230 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.772319 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.772806 (+ 487us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.772898 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251023 20:17:50.774348 941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e601c89a251b4d228dbb0d4dab079b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.774482 941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:50.774828 972 sys_catalog.cc:455] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e601c89a251b4d228dbb0d4dab079b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.774952 972 sys_catalog.cc:458] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:50.775043 978 mvcc.cc:204] Tried to move back new op lower bound from 7214082747437142016 to 7214082747404697600. Current Snapshot: MvccSnapshot[applied={T|T < 7214082747437142016}]
I20251023 20:17:50.775143 979 mvcc.cc:204] Tried to move back new op lower bound from 7214082747437142016 to 7214082747404697600. Current Snapshot: MvccSnapshot[applied={T|T < 7214082747437142016}]
I20251023 20:17:50.777292 955 sys_catalog.cc:455] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e601c89a251b4d228dbb0d4dab079b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.777434 955 sys_catalog.cc:458] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d [sys.catalog]: This master's current role is: LEADER
I20251023 20:17:50.778033 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e601c89a251b4d228dbb0d4dab079b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e601c89a251b4d228dbb0d4dab079b7d" member_type: VOTER last_known_addr { host: "127.9.117.254" port: 45295 } } peers { permanent_uuid: "4c7aa04201f14130b97531bd4d75719f" member_type: VOTER last_known_addr { host: "127.9.117.253" port: 39229 } } peers { permanent_uuid: "fbc722591ec44cc4ad46f312cacfc686" member_type: VOTER last_known_addr { host: "127.9.117.252" port: 37121 } } }
I20251023 20:17:50.778143 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:17:50.778461 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:50.754541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.9.117.252:37121)
1023 20:17:50.754584 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.754586 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.757409 (+ 2823us) server_negotiation.cc:408] Connection header received
1023 20:17:50.764917 (+ 7508us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.764918 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.764931 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.764962 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.768398 (+ 3436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.769135 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:50.769944 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:50.770165 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:50.772354 (+ 2189us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1023 20:17:50.772359 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1023 20:17:50.772360 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1023 20:17:50.772370 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1023 20:17:50.774533 (+ 2163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.777061 (+ 2528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.777063 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.777063 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.777105 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1023 20:17:50.777358 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1023 20:17:50.777359 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1023 20:17:50.777359 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1023 20:17:50.777512 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1023 20:17:50.777605 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:50.777996 (+ 391us) server_negotiation.cc:300] Negotiation successful
1023 20:17:50.778117 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:17:50.778586 981 mvcc.cc:204] Tried to move back new op lower bound from 7214082747437142016 to 7214082747404697600. Current Snapshot: MvccSnapshot[applied={T|T < 7214082747437142016}]
I20251023 20:17:50.779874 974 catalog_manager.cc:1357] Generated new cluster ID: e60258e4578a4d91be4627e46bb2e77b
I20251023 20:17:50.779944 974 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:17:50.797031 974 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:17:50.798149 974 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:17:50.813378 974 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e601c89a251b4d228dbb0d4dab079b7d: Generated new TSK 0
I20251023 20:17:50.813692 974 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251023 20:17:50.837291 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.835722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.9.117.253:39229)
1023 20:17:50.835770 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.835773 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.836511 (+ 738us) server_negotiation.cc:408] Connection header received
1023 20:17:50.836629 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.836631 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.836643 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.836675 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.837213 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35528: BlockingRecv error: recv got EOF from 127.0.0.1:35528 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251023 20:17:50.838587 769 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.836181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.9.117.254:45295)
1023 20:17:50.836227 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.836229 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.838015 (+ 1786us) server_negotiation.cc:408] Connection header received
1023 20:17:50.838122 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.838124 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.838136 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.838168 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.838521 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42876: BlockingRecv error: recv got EOF from 127.0.0.1:42876 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20251023 20:17:50.840232 923 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.839263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.9.117.252:37121)
1023 20:17:50.839314 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.839316 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.839446 (+ 130us) server_negotiation.cc:408] Connection header received
1023 20:17:50.839546 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.839547 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.839587 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.839618 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.840167 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43336: BlockingRecv error: recv got EOF from 127.0.0.1:43336 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20251023 20:17:50.843178 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.842235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.9.117.253:39229)
1023 20:17:50.842280 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.842282 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.842539 (+ 257us) server_negotiation.cc:408] Connection header received
1023 20:17:50.842692 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.842693 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.842716 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.842749 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.843116 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35544: BlockingRecv error: recv got EOF from 127.0.0.1:35544 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20251023 20:17:50.846565 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.845702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.9.117.253:39229)
1023 20:17:50.845742 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.845744 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.846018 (+ 274us) server_negotiation.cc:408] Connection header received
1023 20:17:50.846122 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.846124 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.846134 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.846165 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.846498 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35552: BlockingRecv error: recv got EOF from 127.0.0.1:35552 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20251023 20:17:50.850782 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.849887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.9.117.253:39229)
1023 20:17:50.849944 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.849946 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.850215 (+ 269us) server_negotiation.cc:408] Connection header received
1023 20:17:50.850324 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.850325 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.850337 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.850368 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.850724 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35564: BlockingRecv error: recv got EOF from 127.0.0.1:35564 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20251023 20:17:50.857270 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.856181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.9.117.253:39229)
1023 20:17:50.856224 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.856226 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.856523 (+ 297us) server_negotiation.cc:408] Connection header received
1023 20:17:50.856648 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.856650 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.856663 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.856702 (+ 39us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.857207 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35574: BlockingRecv error: recv got EOF from 127.0.0.1:35574 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20251023 20:17:50.867889 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.866860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.9.117.253:39229)
1023 20:17:50.866923 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.866926 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.867231 (+ 305us) server_negotiation.cc:408] Connection header received
1023 20:17:50.867345 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.867346 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.867360 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.867398 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.867819 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35584: BlockingRecv error: recv got EOF from 127.0.0.1:35584 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20251023 20:17:50.886765 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.885604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.9.117.253:39229)
1023 20:17:50.885659 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.885662 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.886033 (+ 371us) server_negotiation.cc:408] Connection header received
1023 20:17:50.886159 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.886160 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.886174 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.886210 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.886682 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35592: BlockingRecv error: recv got EOF from 127.0.0.1:35592 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20251023 20:17:50.921648 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.920467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.9.117.253:39229)
1023 20:17:50.920537 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.920540 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.920922 (+ 382us) server_negotiation.cc:408] Connection header received
1023 20:17:50.921062 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.921064 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.921077 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.921110 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.921580 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35594: BlockingRecv error: recv got EOF from 127.0.0.1:35594 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20251023 20:17:50.988131 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:50.987359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.9.117.253:39229)
1023 20:17:50.987416 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:50.987419 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:50.987536 (+ 117us) server_negotiation.cc:408] Connection header received
1023 20:17:50.987615 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:50.987616 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:50.987630 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:50.987689 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:50.988047 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35600: BlockingRecv error: recv got EOF from 127.0.0.1:35600 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20251023 20:17:51.118544 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:51.117854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.9.117.253:39229)
1023 20:17:51.117928 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:51.117931 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:51.117942 (+ 11us) server_negotiation.cc:408] Connection header received
1023 20:17:51.118001 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:51.118003 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:51.118018 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:51.118071 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:51.118436 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35608: BlockingRecv error: recv got EOF from 127.0.0.1:35608 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20251023 20:17:51.376901 847 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:17:51.376223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.9.117.253:39229)
1023 20:17:51.376307 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:51.376310 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:51.376320 (+ 10us) server_negotiation.cc:408] Connection header received
1023 20:17:51.376350 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:51.376352 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:51.376366 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:51.376424 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1023 20:17:51.376832 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35612: BlockingRecv error: recv got EOF from 127.0.0.1:35612 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20251023 20:17:51.738974 939 catalog_manager.cc:1269] Loaded cluster ID: e60258e4578a4d91be4627e46bb2e77b
I20251023 20:17:51.739061 939 catalog_manager.cc:1562] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686: loading cluster ID for follower catalog manager: success
I20251023 20:17:51.740073 939 catalog_manager.cc:1584] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686: acquiring CA information for follower catalog manager: success
I20251023 20:17:51.740371 939 catalog_manager.cc:1612] T 00000000000000000000000000000000 P fbc722591ec44cc4ad46f312cacfc686: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:17:51.742100 952 catalog_manager.cc:1269] Loaded cluster ID: e60258e4578a4d91be4627e46bb2e77b
I20251023 20:17:51.742174 952 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f: loading cluster ID for follower catalog manager: success
I20251023 20:17:51.743062 952 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f: acquiring CA information for follower catalog manager: success
I20251023 20:17:51.743330 952 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 4c7aa04201f14130b97531bd4d75719f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:17:51.892802 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:51.890284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.9.117.253:39229)
1023 20:17:51.890423 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:51.890425 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:51.890437 (+ 12us) server_negotiation.cc:408] Connection header received
1023 20:17:51.890507 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:51.890508 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:51.890567 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:51.890749 (+ 182us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:51.891042 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.891577 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:51.892226 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.892364 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:51.892409 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:51.892527 (+ 118us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:51.892598 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:51.892637 (+ 39us) server_negotiation.cc:300] Negotiation successful
1023 20:17:51.892659 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20251023 20:17:51.897868 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:51.895588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.9.117.252:37121)
1023 20:17:51.895720 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:51.895722 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:51.895729 (+ 7us) server_negotiation.cc:408] Connection header received
1023 20:17:51.895781 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:51.895782 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:51.895810 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:51.895877 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:51.896143 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.896665 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:51.897259 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.897431 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:51.897480 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:51.897595 (+ 115us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:51.897673 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:51.897734 (+ 61us) server_negotiation.cc:300] Negotiation successful
1023 20:17:51.897767 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20251023 20:17:51.902495 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:51.899531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.9.117.253:39229)
1023 20:17:51.899578 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:51.899580 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:51.899912 (+ 332us) server_negotiation.cc:408] Connection header received
1023 20:17:51.900005 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:51.900006 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:51.900022 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:51.900078 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:51.900328 (+ 250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.901075 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:51.901969 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.902170 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:51.902195 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:51.902230 (+ 35us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:51.902295 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:51.902376 (+ 81us) server_negotiation.cc:300] Negotiation successful
1023 20:17:51.902407 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251023 20:17:51.902853 1063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:51.899567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.9.117.254:45295)
1023 20:17:51.899783 (+ 216us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:51.899785 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:51.899876 (+ 91us) server_negotiation.cc:408] Connection header received
1023 20:17:51.899973 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:51.899974 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:51.900009 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:51.900097 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:51.900534 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.901314 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:51.902094 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.902241 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:51.902298 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:51.902438 (+ 140us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:51.902505 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:51.902567 (+ 62us) server_negotiation.cc:300] Negotiation successful
1023 20:17:51.902593 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":57,"threads_started":1}
I20251023 20:17:51.903081 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1023 20:17:51.899754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.9.117.252:37121)
1023 20:17:51.899807 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1023 20:17:51.899810 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1023 20:17:51.899906 (+ 96us) server_negotiation.cc:408] Connection header received
1023 20:17:51.900032 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1023 20:17:51.900034 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1023 20:17:51.900049 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1023 20:17:51.900096 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
1023 20:17:51.900519 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.901240 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1023 20:17:51.902073 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1023 20:17:51.902254 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1023 20:17:51.902279 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1023 20:17:51.902310 (+ 31us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1023 20:17:51.902366 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1023 20:17:51.902991 (+ 625us) server_negotiation.cc:300] Negotiation successful
1023 20:17:51.903024 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251023 20:17:51.904888 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 698
I20251023 20:17:51.910815 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 775
I20251023 20:17:51.916550 9687 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskut_hJr/build/release/bin/kudu with pid 852
2025-10-23T20:17:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3964 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7940 ms total)
[----------] Global test environment tear-down
[==========] 57 tests from 4 test suites ran. (371798 ms total)
[ PASSED ] 56 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
1 FAILED TEST